The Python-based tokenizer was built with regular expressions, making support for the new f-string syntax like nested quotes very tricky to get right. In all likelihood, we would end up with ...
# Copyright 2024 TF.Text Authors. # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License ...