648 lines
30 KiB
Rust
648 lines
30 KiB
Rust
/*!
|
|
This crate exposes a variety of regex engines used by the `regex` crate.
|
|
It provides a vast, sprawling and "expert" level API to each regex engine.
|
|
The regex engines provided by this crate focus heavily on finite automata
|
|
implementations and specifically guarantee worst case `O(m * n)` time
|
|
complexity for all searches. (Where `m ~ len(regex)` and `n ~ len(haystack)`.)
|
|
|
|
The primary goal of this crate is to serve as an implementation detail for the
|
|
`regex` crate. A secondary goal is to make its internals available for use by
|
|
others.
|
|
|
|
# Table of contents
|
|
|
|
* [Should I be using this crate?](#should-i-be-using-this-crate) gives some
|
|
reasons for and against using this crate.
|
|
* [Examples](#examples) provides a small selection of things you can do with
|
|
this crate.
|
|
* [Available regex engines](#available-regex-engines) provides a hyperlinked
|
|
list of all regex engines in this crate.
|
|
* [API themes](#api-themes) discusses common elements used throughout this
|
|
crate.
|
|
* [Crate features](#crate-features) documents the extensive list of Cargo
|
|
features available.
|
|
|
|
# Should I be using this crate?
|
|
|
|
If you find yourself here because you just want to use regexes, then you should
|
|
first check out whether the [`regex` crate](https://docs.rs/regex) meets
|
|
your needs. It provides a streamlined and difficult-to-misuse API for regex
|
|
searching.
|
|
|
|
If you're here because there is something specific you want to do that can't
|
|
be easily done with `regex` crate, then you are perhaps in the right place.
|
|
It's most likely that the first stop you'll want to make is to explore the
|
|
[`meta` regex APIs](meta). Namely, the `regex` crate is just a light wrapper
|
|
over a [`meta::Regex`], so its API will probably be the easiest to transition
|
|
to. In contrast to the `regex` crate, the `meta::Regex` API supports more
|
|
search parameters and does multi-pattern searches. However, it isn't quite as
|
|
ergonomic.
|
|
|
|
Otherwise, the following is an inexhaustive list of reasons to use this crate:
|
|
|
|
* You want to analyze or use a [Thompson `NFA`](nfa::thompson::NFA) directly.
|
|
* You want more powerful multi-pattern search than what is provided by
|
|
`RegexSet` in the `regex` crate. All regex engines in this crate support
|
|
multi-pattern searches.
|
|
* You want to use one of the `regex` crate's internal engines directly because
|
|
of some interesting configuration that isn't possible via the `regex` crate.
|
|
For example, a [lazy DFA's configuration](hybrid::dfa::Config) exposes a
|
|
dizzying number of options for controlling its execution.
|
|
* You want to use the lower level search APIs. For example, both the [lazy
|
|
DFA](hybrid::dfa) and [fully compiled DFAs](dfa) support searching by exploring
|
|
the automaton one state at a time. This might be useful, for example, for
|
|
stream searches or searches of strings stored in non-contiguous in memory.
|
|
* You want to build a fully compiled DFA and then [use zero-copy
|
|
deserialization](dfa::dense::DFA::from_bytes) to load it into memory and use
|
|
it for searching. This use case is supported in core-only no-std/no-alloc
|
|
environments.
|
|
* You want to run [anchored searches](Input::anchored) without using the `^`
|
|
anchor in your regex pattern.
|
|
* You need to work-around contention issues with
|
|
sharing a regex across multiple threads. The
|
|
[`meta::Regex::search_with`](meta::Regex::search_with) API permits bypassing
|
|
any kind of synchronization at all by requiring the caller to provide the
|
|
mutable scratch spaced needed during a search.
|
|
* You want to build your own regex engine on top of the `regex` crate's
|
|
infrastructure.
|
|
|
|
# Examples
|
|
|
|
This section tries to identify a few interesting things you can do with this
|
|
crate and demonstrates them.
|
|
|
|
### Multi-pattern searches with capture groups
|
|
|
|
One of the more frustrating limitations of `RegexSet` in the `regex` crate
|
|
(at the time of writing) is that it doesn't report match positions. With this
|
|
crate, multi-pattern support was intentionally designed in from the beginning,
|
|
which means it works in all regex engines and even for capture groups as well.
|
|
|
|
This example shows how to search for matches of multiple regexes, where each
|
|
regex uses the same capture group names to parse different key-value formats.
|
|
|
|
```
|
|
use regex_automata::{meta::Regex, PatternID};
|
|
|
|
let re = Regex::new_many(&[
|
|
r#"(?m)^(?<key>[[:word:]]+)=(?<val>[[:word:]]+)$"#,
|
|
r#"(?m)^(?<key>[[:word:]]+)="(?<val>[^"]+)"$"#,
|
|
r#"(?m)^(?<key>[[:word:]]+)='(?<val>[^']+)'$"#,
|
|
r#"(?m)^(?<key>[[:word:]]+):\s*(?<val>[[:word:]]+)$"#,
|
|
])?;
|
|
let hay = r#"
|
|
best_album="Blow Your Face Out"
|
|
best_quote='"then as it was, then again it will be"'
|
|
best_year=1973
|
|
best_simpsons_episode: HOMR
|
|
"#;
|
|
let mut kvs = vec![];
|
|
for caps in re.captures_iter(hay) {
|
|
// N.B. One could use capture indices '1' and '2' here
|
|
// as well. Capture indices are local to each pattern.
|
|
// (Just like names are.)
|
|
let key = &hay[caps.get_group_by_name("key").unwrap()];
|
|
let val = &hay[caps.get_group_by_name("val").unwrap()];
|
|
kvs.push((key, val));
|
|
}
|
|
assert_eq!(kvs, vec![
|
|
("best_album", "Blow Your Face Out"),
|
|
("best_quote", "\"then as it was, then again it will be\""),
|
|
("best_year", "1973"),
|
|
("best_simpsons_episode", "HOMR"),
|
|
]);
|
|
|
|
# Ok::<(), Box<dyn std::error::Error>>(())
|
|
```
|
|
|
|
### Build a full DFA and walk it manually
|
|
|
|
One of the regex engines in this crate is a fully compiled DFA. It takes worst
|
|
case exponential time to build, but once built, it can be easily explored and
|
|
used for searches. Here's a simple example that uses its lower level APIs to
|
|
implement a simple anchored search by hand.
|
|
|
|
```
|
|
use regex_automata::{dfa::{Automaton, dense}, Input};
|
|
|
|
let dfa = dense::DFA::new(r"(?-u)\b[A-Z]\w+z\b")?;
|
|
let haystack = "Quartz";
|
|
|
|
// The start state is determined by inspecting the position and the
|
|
// initial bytes of the haystack.
|
|
let mut state = dfa.start_state_forward(&Input::new(haystack))?;
|
|
// Walk all the bytes in the haystack.
|
|
for &b in haystack.as_bytes().iter() {
|
|
state = dfa.next_state(state, b);
|
|
}
|
|
// DFAs in this crate require an explicit
|
|
// end-of-input transition if a search reaches
|
|
// the end of a haystack.
|
|
state = dfa.next_eoi_state(state);
|
|
assert!(dfa.is_match_state(state));
|
|
|
|
# Ok::<(), Box<dyn std::error::Error>>(())
|
|
```
|
|
|
|
Or do the same with a lazy DFA that avoids exponential worst case compile time,
|
|
but requires mutable scratch space to lazily build the DFA during the search.
|
|
|
|
```
|
|
use regex_automata::{hybrid::dfa::DFA, Input};
|
|
|
|
let dfa = DFA::new(r"(?-u)\b[A-Z]\w+z\b")?;
|
|
let mut cache = dfa.create_cache();
|
|
let hay = "Quartz";
|
|
|
|
// The start state is determined by inspecting the position and the
|
|
// initial bytes of the haystack.
|
|
let mut state = dfa.start_state_forward(&mut cache, &Input::new(hay))?;
|
|
// Walk all the bytes in the haystack.
|
|
for &b in hay.as_bytes().iter() {
|
|
state = dfa.next_state(&mut cache, state, b)?;
|
|
}
|
|
// DFAs in this crate require an explicit
|
|
// end-of-input transition if a search reaches
|
|
// the end of a haystack.
|
|
state = dfa.next_eoi_state(&mut cache, state)?;
|
|
assert!(state.is_match());
|
|
|
|
# Ok::<(), Box<dyn std::error::Error>>(())
|
|
```
|
|
|
|
### Find all overlapping matches
|
|
|
|
This example shows how to build a DFA and use it to find all possible matches,
|
|
including overlapping matches. A similar example will work with a lazy DFA as
|
|
well. This also works with multiple patterns and will report all matches at the
|
|
same position where multiple patterns match.
|
|
|
|
```
|
|
use regex_automata::{
|
|
dfa::{dense, Automaton, OverlappingState},
|
|
Input, MatchKind,
|
|
};
|
|
|
|
let dfa = dense::DFA::builder()
|
|
.configure(dense::DFA::config().match_kind(MatchKind::All))
|
|
.build(r"(?-u)\w{3,}")?;
|
|
let input = Input::new("homer marge bart lisa maggie");
|
|
let mut state = OverlappingState::start();
|
|
|
|
let mut matches = vec![];
|
|
while let Some(hm) = {
|
|
dfa.try_search_overlapping_fwd(&input, &mut state)?;
|
|
state.get_match()
|
|
} {
|
|
matches.push(hm.offset());
|
|
}
|
|
assert_eq!(matches, vec![
|
|
3, 4, 5, // hom, home, homer
|
|
9, 10, 11, // mar, marg, marge
|
|
15, 16, // bar, bart
|
|
20, 21, // lis, lisa
|
|
25, 26, 27, 28, // mag, magg, maggi, maggie
|
|
]);
|
|
|
|
# Ok::<(), Box<dyn std::error::Error>>(())
|
|
```
|
|
|
|
# Available regex engines
|
|
|
|
The following is a complete list of all regex engines provided by this crate,
|
|
along with a very brief description of it and why you might want to use it.
|
|
|
|
* [`dfa::regex::Regex`] is a regex engine that works on top of either
|
|
[dense](dfa::dense) or [sparse](dfa::sparse) fully compiled DFAs. You might
|
|
use a DFA if you need the fastest possible regex engine in this crate and can
|
|
afford the exorbitant memory usage usually required by DFAs. Low level APIs on
|
|
fully compiled DFAs are provided by the [`Automaton` trait](dfa::Automaton).
|
|
Fully compiled dense DFAs can handle all regexes except for searching a regex
|
|
with a Unicode word boundary on non-ASCII haystacks. A fully compiled DFA based
|
|
regex can only report the start and end of each match.
|
|
* [`hybrid::regex::Regex`] is a regex engine that works on top of a lazily
|
|
built DFA. Its performance profile is very similar to that of fully compiled
|
|
DFAs, but can be slower in some pathological cases. Fully compiled DFAs are
|
|
also amenable to more optimizations, such as state acceleration, that aren't
|
|
available in a lazy DFA. You might use this lazy DFA if you can't abide the
|
|
worst case exponential compile time of a full DFA, but still want the DFA
|
|
search performance in the vast majority of cases. A lazy DFA based regex can
|
|
only report the start and end of each match.
|
|
* [`dfa::onepass::DFA`] is a regex engine that is implemented as a DFA, but
|
|
can report the matches of each capture group in addition to the start and end
|
|
of each match. The catch is that it only works on a somewhat small subset of
|
|
regexes known as "one-pass." You'll want to use this for cases when you need
|
|
capture group matches and the regex is one-pass since it is likely to be faster
|
|
than any alternative. A one-pass DFA can handle all types of regexes, but does
|
|
have some reasonable limits on the number of capture groups it can handle.
|
|
* [`nfa::thompson::backtrack::BoundedBacktracker`] is a regex engine that uses
|
|
backtracking, but keeps track of the work it has done to avoid catastrophic
|
|
backtracking. Like the one-pass DFA, it provides the matches of each capture
|
|
group. It retains the `O(m * n)` worst case time bound. This tends to be slower
|
|
than the one-pass DFA regex engine, but faster than the PikeVM. It can handle
|
|
all types of regexes, but usually only works well with small haystacks and
|
|
small regexes due to the memory required to avoid redoing work.
|
|
* [`nfa::thompson::pikevm::PikeVM`] is a regex engine that can handle all
|
|
regexes, of all sizes and provides capture group matches. It tends to be a tool
|
|
of last resort because it is also usually the slowest regex engine.
|
|
* [`meta::Regex`] is the meta regex engine that combines *all* of the above
|
|
engines into one. The reason for this is that each of the engines above have
|
|
their own caveats such as, "only handles a subset of regexes" or "is generally
|
|
slow." The meta regex engine accounts for all of these caveats and composes
|
|
the engines in a way that attempts to mitigate each engine's weaknesses while
|
|
emphasizing its strengths. For example, it will attempt to run a lazy DFA even
|
|
if it might fail. In which case, it will restart the search with a likely
|
|
slower but more capable regex engine. The meta regex engine is what you should
|
|
default to. Use one of the above engines directly only if you have a specific
|
|
reason to.
|
|
|
|
# API themes
|
|
|
|
While each regex engine has its own APIs and configuration options, there are
|
|
some general themes followed by all of them.
|
|
|
|
### The `Input` abstraction
|
|
|
|
Most search routines in this crate accept anything that implements
|
|
`Into<Input>`. Both `&str` and `&[u8]` haystacks satisfy this constraint, which
|
|
means that things like `engine.search("foo")` will work as you would expect.
|
|
|
|
By virtue of accepting an `Into<Input>` though, callers can provide more than
|
|
just a haystack. Indeed, the [`Input`] type has more details, but briefly,
|
|
callers can use it to configure various aspects of the search:
|
|
|
|
* The span of the haystack to search via [`Input::span`] or [`Input::range`],
|
|
which might be a substring of the haystack.
|
|
* Whether to run an anchored search or not via [`Input::anchored`]. This
|
|
permits one to require matches to start at the same offset that the search
|
|
started.
|
|
* Whether to ask the regex engine to stop as soon as a match is seen via
|
|
[`Input::earliest`]. This can be used to find the offset of a match as soon
|
|
as it is known without waiting for the full leftmost-first match to be found.
|
|
This can also be used to avoid the worst case `O(m * n^2)` time complexity
|
|
of iteration.
|
|
|
|
Some lower level search routines accept an `&Input` for performance reasons.
|
|
In which case, `&Input::new("haystack")` can be used for a simple search.
|
|
|
|
### Error reporting
|
|
|
|
Most, but not all, regex engines in this crate can fail to execute a search.
|
|
When a search fails, callers cannot determine whether or not a match exists.
|
|
That is, the result is indeterminate.
|
|
|
|
Search failure, in all cases in this crate, is represented by a [`MatchError`].
|
|
Routines that can fail start with the `try_` prefix in their name. For example,
|
|
[`hybrid::regex::Regex::try_search`] can fail for a number of reasons.
|
|
Conversely, routines that either can't fail or can panic on failure lack the
|
|
`try_` prefix. For example, [`hybrid::regex::Regex::find`] will panic in
|
|
cases where [`hybrid::regex::Regex::try_search`] would return an error, and
|
|
[`meta::Regex::find`] will never panic. Therefore, callers need to pay close
|
|
attention to the panicking conditions in the documentation.
|
|
|
|
In most cases, the reasons that a search fails are either predictable or
|
|
configurable, albeit at some additional cost.
|
|
|
|
An example of predictable failure is
|
|
[`BoundedBacktracker::try_search`](nfa::thompson::backtrack::BoundedBacktracker::try_search).
|
|
Namely, it fails whenever the multiplication of the haystack, the regex and some
|
|
constant exceeds the
|
|
[configured visited capacity](nfa::thompson::backtrack::Config::visited_capacity).
|
|
Callers can predict the failure in terms of haystack length via the
|
|
[`BoundedBacktracker::max_haystack_len`](nfa::thompson::backtrack::BoundedBacktracker::max_haystack_len)
|
|
method. While this form of failure is technically avoidable by increasing the
|
|
visited capacity, it isn't practical to do so for all inputs because the
|
|
memory usage required for larger haystacks becomes impractically large. So in
|
|
practice, if one is using the bounded backtracker, you really do have to deal
|
|
with the failure.
|
|
|
|
An example of configurable failure happens when one enables heuristic support
|
|
for Unicode word boundaries in a DFA. Namely, since the DFAs in this crate
|
|
(except for the one-pass DFA) do not support Unicode word boundaries on
|
|
non-ASCII haystacks, building a DFA from an NFA that contains a Unicode word
|
|
boundary will itself fail. However, one can configure DFAs to still be built in
|
|
this case by
|
|
[configuring heuristic support for Unicode word boundaries](hybrid::dfa::Config::unicode_word_boundary).
|
|
If the NFA the DFA is built from contains a Unicode word boundary, then the
|
|
DFA will still be built, but special transitions will be added to every state
|
|
that cause the DFA to fail if any non-ASCII byte is seen. This failure happens
|
|
at search time and it requires the caller to opt into this.
|
|
|
|
There are other ways for regex engines to fail in this crate, but the above
|
|
two should represent the general theme of failures one can find. Dealing
|
|
with these failures is, in part, one the responsibilities of the [meta regex
|
|
engine](meta). Notice, for example, that the meta regex engine exposes an API
|
|
that never returns an error nor panics. It carefully manages all of the ways
|
|
in which the regex engines can fail and either avoids the predictable ones
|
|
entirely (e.g., the bounded backtracker) or reacts to configured failures by
|
|
falling back to a different engine (e.g., the lazy DFA quitting because it saw
|
|
a non-ASCII byte).
|
|
|
|
### Configuration and Builders
|
|
|
|
Most of the regex engines in this crate come with two types to facilitate
|
|
building the regex engine: a `Config` and a `Builder`. A `Config` is usually
|
|
specific to that particular regex engine, but other objects such as parsing and
|
|
NFA compilation have `Config` types too. A `Builder` is the thing responsible
|
|
for taking inputs (either pattern strings or already-parsed patterns or even
|
|
NFAs directly) and turning them into an actual regex engine that can be used
|
|
for searching.
|
|
|
|
The main reason why building a regex engine is a bit complicated is because
|
|
of the desire to permit composition with de-coupled components. For example,
|
|
you might want to [manually construct a Thompson NFA](nfa::thompson::Builder)
|
|
and then build a regex engine from it without ever using a regex parser
|
|
at all. On the other hand, you might also want to build a regex engine directly
|
|
from the concrete syntax. This demonstrates why regex engine construction is
|
|
so flexible: it needs to support not just convenient construction, but also
|
|
construction from parts built elsewhere.
|
|
|
|
This is also in turn why there are many different `Config` structs in this
|
|
crate. Let's look more closely at an example: [`hybrid::regex::Builder`]. It
|
|
accepts three different `Config` types for configuring construction of a lazy
|
|
DFA regex:
|
|
|
|
* [`hybrid::regex::Builder::syntax`] accepts a
|
|
[`util::syntax::Config`] for configuring the options found in the
|
|
[`regex-syntax`](regex_syntax) crate. For example, whether to match
|
|
case insensitively.
|
|
* [`hybrid::regex::Builder::thompson`] accepts a [`nfa::thompson::Config`] for
|
|
configuring construction of a [Thompson NFA](nfa::thompson::NFA). For example,
|
|
whether to build an NFA that matches the reverse language described by the
|
|
regex.
|
|
* [`hybrid::regex::Builder::dfa`] accept a [`hybrid::dfa::Config`] for
|
|
configuring construction of the pair of underlying lazy DFAs that make up the
|
|
lazy DFA regex engine. For example, changing the capacity of the cache used to
|
|
store the transition table.
|
|
|
|
The lazy DFA regex engine uses all three of those configuration objects for
|
|
methods like [`hybrid::regex::Builder::build`], which accepts a pattern
|
|
string containing the concrete syntax of your regex. It uses the syntax
|
|
configuration to parse it into an AST and translate it into an HIR. Then the
|
|
NFA configuration when compiling the HIR into an NFA. And then finally the DFA
|
|
configuration when lazily determinizing the NFA into a DFA.
|
|
|
|
Notice though that the builder also has a
|
|
[`hybrid::regex::Builder::build_from_dfas`] constructor. This permits callers
|
|
to build the underlying pair of lazy DFAs themselves (one for the forward
|
|
searching to find the end of a match and one for the reverse searching to find
|
|
the start of a match), and then build the regex engine from them. The lazy
|
|
DFAs, in turn, have their own builder that permits [construction directly from
|
|
a Thompson NFA](hybrid::dfa::Builder::build_from_nfa). Continuing down the
|
|
rabbit hole, a Thompson NFA has its own compiler that permits [construction
|
|
directly from an HIR](nfa::thompson::Compiler::build_from_hir). The lazy DFA
|
|
regex engine builder lets you follow this rabbit hole all the way down, but
|
|
also provides convenience routines that do it for you when you don't need
|
|
precise control over every component.
|
|
|
|
The [meta regex engine](meta) is a good example of something that utilizes the
|
|
full flexibility of these builders. It often needs not only precise control
|
|
over each component, but also shares them across multiple regex engines.
|
|
(Most sharing is done by internal reference accounting. For example, an
|
|
[`NFA`](nfa::thompson::NFA) is reference counted internally which makes cloning
|
|
cheap.)
|
|
|
|
### Size limits
|
|
|
|
Unlike the `regex` crate, the `regex-automata` crate specifically does not
|
|
enable any size limits by default. That means users of this crate need to
|
|
be quite careful when using untrusted patterns. Namely, because bounded
|
|
repetitions can grow exponentially by stacking them, it is possible to build a
|
|
very large internal regex object from just a small pattern string. For example,
|
|
the NFA built from the pattern `a{10}{10}{10}{10}{10}{10}{10}` is over 240MB.
|
|
|
|
There are multiple size limit options in this crate. If one or more size limits
|
|
are relevant for the object you're building, they will be configurable via
|
|
methods on a corresponding `Config` type.
|
|
|
|
# Crate features
|
|
|
|
This crate has a dizzying number of features. The main idea is to be able to
|
|
control how much stuff you pull in for your specific use case, since the full
|
|
crate is quite large and can dramatically increase compile times and binary
|
|
size.
|
|
|
|
The most barebones but useful configuration is to disable all default features
|
|
and enable only `dfa-search`. This will bring in just the DFA deserialization
|
|
and search routines without any dependency on `std` or `alloc`. This does
|
|
require generating and serializing a DFA, and then storing it somewhere, but
|
|
it permits regex searches in freestanding or embedded environments.
|
|
|
|
Because there are so many features, they are split into a few groups.
|
|
|
|
The default set of features is: `std`, `syntax`, `perf`, `unicode`, `meta`,
|
|
`nfa`, `dfa` and `hybrid`. Basically, the default is to enable everything
|
|
except for development related features like `logging`.
|
|
|
|
### Ecosystem features
|
|
|
|
* **std** - Enables use of the standard library. In terms of APIs, this usually
|
|
just means that error types implement the `std::error::Error` trait. Otherwise,
|
|
`std` sometimes enables the code to be faster, for example, using a `HashMap`
|
|
instead of a `BTreeMap`. (The `std` feature matters more for dependencies like
|
|
`aho-corasick` and `memchr`, where `std` is required to enable certain classes
|
|
of SIMD optimizations.) Enabling `std` automatically enables `alloc`.
|
|
* **alloc** - Enables use of the `alloc` library. This is required for most
|
|
APIs in this crate. The main exception is deserializing and searching with
|
|
fully compiled DFAs.
|
|
* **logging** - Adds a dependency on the `log` crate and makes this crate emit
|
|
log messages of varying degrees of utility. The log messages are especially
|
|
useful in trying to understand what the meta regex engine is doing.
|
|
|
|
### Performance features
|
|
|
|
* **perf** - Enables all of the below features.
|
|
* **perf-inline** - When enabled, `inline(always)` is used in (many) strategic
|
|
locations to help performance at the expense of longer compile times and
|
|
increased binary size.
|
|
* **perf-literal** - Enables all literal related optimizations.
|
|
* **perf-literal-substring** - Enables all single substring literal
|
|
optimizations. This includes adding a dependency on the `memchr` crate.
|
|
* **perf-literal-multisubstring** - Enables all multiple substring literal
|
|
optimizations. This includes adding a dependency on the `aho-corasick`
|
|
crate.
|
|
|
|
### Unicode features
|
|
|
|
* **unicode** -
|
|
Enables all Unicode features. This feature is enabled by default, and will
|
|
always cover all Unicode features, even if more are added in the future.
|
|
* **unicode-age** -
|
|
Provide the data for the
|
|
[Unicode `Age` property](https://www.unicode.org/reports/tr44/tr44-24.html#Character_Age).
|
|
This makes it possible to use classes like `\p{Age:6.0}` to refer to all
|
|
codepoints first introduced in Unicode 6.0
|
|
* **unicode-bool** -
|
|
Provide the data for numerous Unicode boolean properties. The full list
|
|
is not included here, but contains properties like `Alphabetic`, `Emoji`,
|
|
`Lowercase`, `Math`, `Uppercase` and `White_Space`.
|
|
* **unicode-case** -
|
|
Provide the data for case insensitive matching using
|
|
[Unicode's "simple loose matches" specification](https://www.unicode.org/reports/tr18/#Simple_Loose_Matches).
|
|
* **unicode-gencat** -
|
|
Provide the data for
|
|
[Unicode general categories](https://www.unicode.org/reports/tr44/tr44-24.html#General_Category_Values).
|
|
This includes, but is not limited to, `Decimal_Number`, `Letter`,
|
|
`Math_Symbol`, `Number` and `Punctuation`.
|
|
* **unicode-perl** -
|
|
Provide the data for supporting the Unicode-aware Perl character classes,
|
|
corresponding to `\w`, `\s` and `\d`. This is also necessary for using
|
|
Unicode-aware word boundary assertions. Note that if this feature is
|
|
disabled, the `\s` and `\d` character classes are still available if the
|
|
`unicode-bool` and `unicode-gencat` features are enabled, respectively.
|
|
* **unicode-script** -
|
|
Provide the data for
|
|
[Unicode scripts and script extensions](https://www.unicode.org/reports/tr24/).
|
|
This includes, but is not limited to, `Arabic`, `Cyrillic`, `Hebrew`,
|
|
`Latin` and `Thai`.
|
|
* **unicode-segment** -
|
|
Provide the data necessary to provide the properties used to implement the
|
|
[Unicode text segmentation algorithms](https://www.unicode.org/reports/tr29/).
|
|
This enables using classes like `\p{gcb=Extend}`, `\p{wb=Katakana}` and
|
|
`\p{sb=ATerm}`.
|
|
* **unicode-word-boundary** -
|
|
Enables support for Unicode word boundaries, i.e., `\b`, in regexes. When
|
|
this and `unicode-perl` are enabled, then data tables from `regex-syntax` are
|
|
used to implement Unicode word boundaries. However, if `regex-syntax` isn't
|
|
enabled as a dependency then one can still enable this feature. It will
|
|
cause `regex-automata` to bundle its own data table that would otherwise be
|
|
redundant with `regex-syntax`'s table.
|
|
|
|
### Regex engine features
|
|
|
|
* **syntax** - Enables a dependency on `regex-syntax`. This makes APIs
|
|
for building regex engines from pattern strings available. Without the
|
|
`regex-syntax` dependency, the only way to build a regex engine is generally
|
|
to deserialize a previously built DFA or to hand assemble an NFA using its
|
|
[builder API](nfa::thompson::Builder). Once you have an NFA, you can build any
|
|
of the regex engines in this crate. The `syntax` feature also enables `alloc`.
|
|
* **meta** - Enables the meta regex engine. This also enables the `syntax` and
|
|
`nfa-pikevm` features, as both are the minimal requirements needed. The meta
|
|
regex engine benefits from enabling any of the other regex engines and will
|
|
use them automatically when appropriate.
|
|
* **nfa** - Enables all NFA related features below.
|
|
* **nfa-thompson** - Enables the Thompson NFA APIs. This enables `alloc`.
|
|
* **nfa-pikevm** - Enables the PikeVM regex engine. This enables
|
|
`nfa-thompson`.
|
|
* **nfa-backtrack** - Enables the bounded backtracker regex engine. This
|
|
enables `nfa-thompson`.
|
|
* **dfa** - Enables all DFA related features below.
|
|
* **dfa-build** - Enables APIs for determinizing DFAs from NFAs. This
|
|
enables `nfa-thompson` and `dfa-search`.
|
|
* **dfa-search** - Enables APIs for searching with DFAs.
|
|
* **dfa-onepass** - Enables the one-pass DFA API. This enables
|
|
`nfa-thompson`.
|
|
* **hybrid** - Enables the hybrid NFA/DFA or "lazy DFA" regex engine. This
|
|
enables `alloc` and `nfa-thompson`.
|
|
|
|
*/
|
|
|
|
// We are no_std.
|
|
#![no_std]
|
|
// All APIs need docs!
|
|
#![deny(missing_docs)]
|
|
// Some intra-doc links are broken when certain features are disabled, so we
|
|
// only bleat about it when most (all?) features are enabled. But when we do,
|
|
// we block the build. Links need to work.
|
|
#![cfg_attr(
|
|
all(
|
|
feature = "std",
|
|
feature = "nfa",
|
|
feature = "dfa",
|
|
feature = "hybrid"
|
|
),
|
|
deny(rustdoc::broken_intra_doc_links)
|
|
)]
|
|
// Broken rustdoc links are very easy to come by when you start disabling
|
|
// features. Namely, features tend to change imports, and imports change what's
|
|
// available to link to.
|
|
//
|
|
// Basically, we just don't support rustdoc for anything other than the maximal
|
|
// feature configuration. Other configurations will work, they just won't be
|
|
// perfect.
|
|
//
|
|
// So here, we specifically allow them so we don't even get warned about them.
|
|
#![cfg_attr(
|
|
not(all(
|
|
feature = "std",
|
|
feature = "nfa",
|
|
feature = "dfa",
|
|
feature = "hybrid"
|
|
)),
|
|
allow(rustdoc::broken_intra_doc_links)
|
|
)]
|
|
// Kinda similar, but eliminating all of the dead code and unused import
|
|
// warnings for every feature combo is a fool's errand. Instead, we just
|
|
// suppress those, but still let them through in a common configuration when we
|
|
// build most of everything.
|
|
//
|
|
// This does actually suggest that when features are disabled, we are actually
|
|
// compiling more code than we need to be. And this is perhaps not so great
|
|
// because disabling features is usually done in order to reduce compile times
|
|
// by reducing the amount of code one compiles... However, usually, most of the
|
|
// time this dead code is a relatively small amount from the 'util' module.
|
|
// But... I confess... There isn't a ton of visibility on this.
|
|
//
|
|
// I'm happy to try to address this in a different way, but "let's annotate
|
|
// every function in 'util' with some non-local combination of features" just
|
|
// cannot be the way forward.
|
|
#![cfg_attr(
|
|
not(all(
|
|
feature = "std",
|
|
feature = "nfa",
|
|
feature = "dfa",
|
|
feature = "hybrid",
|
|
feature = "perf-literal-substring",
|
|
feature = "perf-literal-multisubstring",
|
|
)),
|
|
allow(dead_code, unused_imports, unused_variables)
|
|
)]
|
|
// We generally want all types to impl Debug.
|
|
#![warn(missing_debug_implementations)]
|
|
// No clue why this thing is still unstable because it's pretty amazing. This
|
|
// adds Cargo feature annotations to items in the rustdoc output. Which is
|
|
// sadly hugely beneficial for this crate due to the number of features.
|
|
#![cfg_attr(docsrs, feature(doc_auto_cfg))]
|
|
|
|
// I have literally never tested this crate on 16-bit, so it is quite
|
|
// suspicious to advertise support for it. But... the regex crate, at time
|
|
// of writing, at least claims to support it by not doing any conditional
|
|
// compilation based on the target pointer width. So I guess I remain
|
|
// consistent with that here.
|
|
//
|
|
// If you are here because you're on a 16-bit system and you were somehow using
|
|
// the regex crate previously, please file an issue. Please be prepared to
|
|
// provide some kind of reproduction or carve out some path to getting 16-bit
|
|
// working in CI. (Via qemu?)
|
|
#[cfg(not(any(
|
|
target_pointer_width = "16",
|
|
target_pointer_width = "32",
|
|
target_pointer_width = "64"
|
|
)))]
|
|
compile_error!("not supported on non-{16,32,64}, please file an issue");
|
|
|
|
#[cfg(any(test, feature = "std"))]
|
|
extern crate std;
|
|
|
|
#[cfg(feature = "alloc")]
|
|
extern crate alloc;
|
|
|
|
#[cfg(doctest)]
|
|
doc_comment::doctest!("../README.md");
|
|
|
|
#[doc(inline)]
|
|
pub use crate::util::primitives::PatternID;
|
|
pub use crate::util::search::*;
|
|
|
|
#[macro_use]
|
|
mod macros;
|
|
|
|
#[cfg(any(feature = "dfa-search", feature = "dfa-onepass"))]
|
|
pub mod dfa;
|
|
#[cfg(feature = "hybrid")]
|
|
pub mod hybrid;
|
|
#[cfg(feature = "meta")]
|
|
pub mod meta;
|
|
#[cfg(feature = "nfa-thompson")]
|
|
pub mod nfa;
|
|
pub mod util;
|