Skip to content

Commit b603f19

Browse files
tf-text-github-robottf-text-github-robot
committed
Fixed a bug in FastWordpieceTokenizer where vocab sizes >= 7 would cause failures if the unknown token was not at the end of the vocabulary by ensuring the internal hash map is reserved upfront.
PiperOrigin-RevId: 870823136
1 parent afdd8f2 commit b603f19

314 files changed

Lines changed: 315 additions & 314 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎oss_scripts/build_docs.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# coding=utf-8
2-
# Copyright 2025 TF.Text Authors.
2+
# Copyright 2026 TF.Text Authors.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.

‎oss_scripts/pip_package/build_pip_package.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# coding=utf-8
2-
# Copyright 2025 TF.Text Authors.
2+
# Copyright 2026 TF.Text Authors.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.

‎oss_scripts/pip_package/setup.nightly.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# coding=utf-8
2-
# Copyright 2025 TF.Text Authors.
2+
# Copyright 2026 TF.Text Authors.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.

‎oss_scripts/pip_package/setup.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# coding=utf-8
2-
# Copyright 2025 TF.Text Authors.
2+
# Copyright 2026 TF.Text Authors.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.

‎oss_scripts/pip_package/tensorflow_build_info.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# coding=utf-8
2-
# Copyright 2025 TF.Text Authors.
2+
# Copyright 2026 TF.Text Authors.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.

‎tensorflow_text/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# coding=utf-8
2-
# Copyright 2025 TF.Text Authors.
2+
# Copyright 2026 TF.Text Authors.
33
#
44
# Licensed under the Apache License, Version 2.0 (the "License");
55
# you may not use this file except in compliance with the License.

‎tensorflow_text/core/kernels/boise_offset_converter.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2025 TF.Text Authors.
1+
// Copyright 2026 TF.Text Authors.
22
//
33
// Licensed under the Apache License, Version 2.0 (the "License");
44
// you may not use this file except in compliance with the License.

‎tensorflow_text/core/kernels/boise_offset_converter.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2025 TF.Text Authors.
1+
// Copyright 2026 TF.Text Authors.
22
//
33
// Licensed under the Apache License, Version 2.0 (the "License");
44
// you may not use this file except in compliance with the License.

‎tensorflow_text/core/kernels/boise_offset_converter_kernel.cc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2025 TF.Text Authors.
1+
// Copyright 2026 TF.Text Authors.
22
//
33
// Licensed under the Apache License, Version 2.0 (the "License");
44
// you may not use this file except in compliance with the License.

‎tensorflow_text/core/kernels/boise_offset_converter_kernel.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
// Copyright 2025 TF.Text Authors.
1+
// Copyright 2026 TF.Text Authors.
22
//
33
// Licensed under the Apache License, Version 2.0 (the "License");
44
// you may not use this file except in compliance with the License.

0 commit comments

Comments
 (0)