A character encoding detector for legacy Web content.
Please see the file named COPYRIGHT.
Generated API documentation is available online.
There is a long-form write-up about the design and motivation of the crate.
The purpose of this detector is user retention for Firefox by ensuring that the long tail of the legacy Web is not more convenient to use in Chrome than in Firefox. (Chrome deployed ced, which left Firefox less convenient to use until the deployment of this detector.)
chardet was the name of Mozilla's old encoding detector. I named this one
chardetng, because this the next generation of encoding detector in Firefox. There is no code reuse from the old
This crate aims to be more accurate than ICU, more complete than
chardet, more explainable and modifiable than
compact_enc_det (aka. ced), and, in an application that already depends on
encoding_rs for other reasons, smaller in added binary footprint than
Enabling the optional feature
chardetng run the detectors for individual encodings in parallel. Unfortunately, the performance doesn't scale linearly with CPU cores, but it's still better than single-threaded performance in terms of wall-clock time if a single instance of
chardetng is running. In terms of combined CPU core usage, the
multithreading mode is quite a bit worse than the single-threaded more, so if you can find a parallelization point at some higher-level task such that you could have multiple instances of
chardetng running in paraller each on a single thread, you'll get better results doing that.
chardetng prefers to do negative matching (rule out possibilities from the set of plausible encodings) than to do positive matching. Since negative matching is insufficient, there is positive matching, too.
Detected only if explicitly permitted by the argument to the `guess` method. It's harmful for Web browsers to detect UTF-8 without requiring user action, such as choosing a menu item, because Web developers would start relying on the detection.
Not detected: Detecting these belongs on the BOM layer.
Not detected: This encoding is for XHR.
<meta charset=x-user-defined> in HTML is not unlabeled and means windows-1252.
Detected as GBK.
Detected: Historical locale-specific fallbacks.
Detected: Detected by multiple browsers past and present.
Detected as KOI8-U. (Always guessing the U variant is less likely to corrupt non-box drawing characters.)
Detected as windows-1255.
Detected: Detected by IE and Chrome; in menu in IE and Firefox.
Detected: Detected by IE and Chrome.
Detected: Available in menu in IE and Firefox.
Detected: Detected by Chrome. This encoding is so similar to windows-1257 that menu items for windows-1257 can be considered to accommodate this one in IE and Firefox. Due to the mechanics of this detector, if this wasn't included as a separate item, the windows-1257 detection wouldn't catch the cases that use curly quotes and are invalid as windows-1257.
Not detected: Not detected by IE and Chrome. (Was previously detected by Firefox.)
Not detected: These encodings have never been a locale-specific fallback in a major browser or a menu item in IE.
feedmethod using Rayon.