Introduction

When we manipulate strings, the Unicode format establishes itself because of its compatibility with historical formats (like ASCII) and its capacity to understand a large range of characters and symbols for all cultures and all regions in the world. PHP provides several tools to manipulate such strings, like the following extensions: mbstring, iconv or also the excellent intl which is based on ICU, the reference implementation of Unicode. Unfortunately, sometimes we have to mix these extensions to achieve our aims and at the cost of a certain complexity along with a regrettable verbosity.

The Hoa\Ustring library answers to these issues by providing a simple way to manipulate strings with performance and efficiency in minds. It also provides some evoluated algorithms to perform search operations on strings.

Unicode strings

The Hoa\Ustring\Ustring class represents a UTF-8 Unicode strings and allows to manipulate it easily. This class implements the ArrayAccess, Countable and IteratorAggregate interfaces. We are going to use three examples in three different languages: French, Arab and Japanese. Thus:

$french   = new Hoa\Ustring\Ustring('Je t\'aime');
$arabic   = new Hoa\Ustring\Ustring('أحبك');
$japanese = new Hoa\Ustring\Ustring('私はあなたを愛して');

Now, let's see what we can do on these three strings.

String manipulation

Let's start with elementary operations. If we would like to count the number of characters (not bytes), we will use the count function. Thus:

var_dump(
    count($french),
    count($arabic),
    count($japanese)
);

/**
 * Will output:
 *     int(9)
 *     int(4)
 *     int(9)
 */

When we speak about text position, it is not suitable to speak about the right or the left, but rather about a beginning or an end, and based on the direction of writing. We can know this direction thanks to the Hoa\Ustring\Ustring::getDirection method. It returns the value of one of the following constants:

• Hoa\Ustring\Ustring::LTR, for left-to-right, if the text is written from the left to the right,
• Hoa\Ustring\Ustring::RTL, for right-to-left, if the text is written from the right to the left.

Let's observe the result with our examples:

var_dump(
    $french->getDirection()   === Hoa\Ustring\Ustring::LTR, // is left-to-right?
    $arabic->getDirection()   === Hoa\Ustring\Ustring::RTL, // is right-to-left?
    $japanese->getDirection() === Hoa\Ustring\Ustring::LTR  // is left-to-right?
);

/**
 * Will output:
 *     bool(true)
 *     bool(true)
 *     bool(true)
 */

The result of this method is computed thanks to the Hoa\Ustring\Ustring::getCharDirection static method which computes the direction of only one character.

If we would like to concatenate another string to the end or to the beginning, we will respectively use the Hoa\Ustring\Ustring::append and Hoa\Ustring\Ustring::prepend methods. These methods, like most of the ones which modifies the string, return the object itself, in order to chain the calls. For instance:

echo $french->append('… et toi, m\'aimes-tu ?')->prepend('Mam\'zelle ! ');

/**
 * Will output:
 *     Mam'zelle ! Je t'aime… et toi, m'aimes-tu ?
 */

We also have the Hoa\Ustring\Ustring::toLowerCase and Hoa\Ustring\Ustring::toUpperCase methods to, respectively, set the case of the string to lower or upper. For instance:

echo $french->toUpperCase();

/**
 * Will output:
 *     MAM'ZELLE ! JE T'AIME… ET TOI, M'AIMES-TU ?
 */

We can also add characters to the beginning or to the end of the string to reach a minimum length. This operation is frequently called the padding (for historical reasons dating back to typewriters). That's why we have the Hoa\Ustring\Ustring::pad method which takes three arguments: the minimum length, characters to add and a constant indicating whether we have to add at the end or at the beginning of the string (respectively Hoa\Ustring\Ustring::END, by default, and Hoa\Ustring\Ustring::BEGINNING).

echo $arabic->pad(20, ' ');

/**
 * Will output:
 *                     أحبك
 */

A similar operation allows to remove, by default, spaces at the beginning and at the end of the string thanks to the Hoa\Ustring\Ustring::trim method. For example, to retreive our original Arabic string:

echo $arabic->trim();

/**
 * Will output:
 *     أحبك
 */

If we would like to remove other characters, we can use its first argument which must be a regular expression. Finally, its second argument allows to specify from what side we would like to remove character: at the beginning, at the end or both, still by using the Hoa\Ustring\Ustring::BEGINNING and Hoa\Ustring\Ustring::END constants.

If we would like to remove other characters, we can use its first argument which must be a regular expression. Finally, its second argument allows to specify the side where to remove characters: at the beginning, at the end or both, still by using the Hoa\Ustring\Ustring::BEGINNING and Hoa\Ustring\Ustring::END constants. We can combine these constants to express “both sides”, which is the default value: Hoa\Ustring\Ustring::BEGINNING | Hoa\Ustring\Ustring::END. For example, to remove all the numbers and the spaces only at the end, we will write:

$arabic->trim('\s|\d', Hoa\Ustring\Ustring::END);

We can also reduce the string to a sub-string by specifying the position of the first character followed by the length of the sub-string to the Hoa\Ustring\Ustring::reduce method:

echo $french->reduce(3, 6)->reduce(2, 4);

/**
 * Will output:
 *     aime
 */

If we would like to get a specific character, we can rely on the ArrayAccess interface. For instance, to get the first character of each of our examples (from their original definitions):

var_dump(
    $french[0],
    $arabic[0],
    $japanese[0]
);

/**
 * Will output:
 *     string(1) "J"
 *     string(2) "أ"
 *     string(3) "私"
 */

If we would like the last character, we will use the -1 index. The index is not bounded to the length of the string. If the index exceeds this length, then a modulo will be applied.

We can also modify or remove a specific character with this method. For example:

$french->append(' ?');
$french[-1] = '!';
echo $french;

/**
 * Will output:
 *     Je t'aime !
 */

Another very useful method is the ASCII transformation. Be careful, this is not always possible, according to your settings. For example:

$title = new Hoa\Ustring\Ustring('Un été brûlant sur la côte');
echo $title->toAscii();

/**
 * Will output:
 *     Un ete brulant sur la cote
 */

We can also transform from Arabic or Japanese to ASCII. Symbols, like Mathemeticals symbols or emojis, are also transformed:

$emoji = new Hoa\Ustring\Ustring('I ❤ Unicode');
$maths = new Hoa\Ustring\Ustring('∀ i ∈ ℕ');

echo
    $arabic->toAscii(), "\n",
    $japanese->toAscii(), "\n",
    $emoji->toAscii(), "\n",
    $maths->toAscii(), "\n";

/**
 * Will output:
 *     ahbk
 *     sihaanatawo aishite
 *     I (heavy black heart)️ Unicode
 *     (for all) i (element of) N
 */

In order this method to work correctly, the intl extension needs to be present, so that the Transliterator class is present. If it does not exist, the Normalizer class must exist. If this class does not exist neither, the Hoa\Ustring\Ustring::toAscii method can still try a transformation, but it is less efficient. To activate this last solution, true must be passed as a single argument. This tour de force is not recommended in most cases.

We also find the getTransliterator method which returns a Transliterator object, or null if this class does not exist. This method takes a transliteration identifier as argument. We suggest to read the documentation about the transliterator of ICU to understand this identifier. The transliterate method allows to transliterate the current string based on an identifier and a beginning index and an end one. This method works the same way than the Transliterator::transliterate method.

More generally, to change the encoding format, we can use the Hoa\Ustring\Ustring::transcode static method, with a string as first argument, the original encoding format as second argument and the expected encoding format as third argument (UTF-8 by default). The get the list of encoding formats, we have to refer to the iconv extension or to use the following command line in a terminal:

$ iconv --list

To know if a string is encoded in UTF-8, we can use the Hoa\Ustring\Ustring::isUtf8 static method; for instance:

var_dump(
    Hoa\Ustring\Ustring::isUtf8('a'),
    Hoa\Ustring\Ustring::isUtf8(Hoa\Ustring\Ustring::transcode('a', 'UTF-8', 'UTF-16'))
);

/**
 * Will output:
 *     bool(true)
 *     bool(false)
 */

We can split the string into several sub-strings by using the Hoa\Ustring\Ustring::split method. As first argument, we have a regular expression (of kind PCRE), then an integer representing the maximum number of elements to return and finally a combination of constants. These constants are the same as the ones of preg_split.

By default, the second argument is set to -1, which means infinity, and the last argument is set to PREG_SPLIT_NO_EMPTY. Thus, if we would like to get all the words of a string, we will write:

print_r($title->split('#\b|\s#'));

/**
 * Will output:
 *     Array
 *     (
 *         [0] => Un
 *         [1] => ete
 *         [2] => brulant
 *         [3] => sur
 *         [4] => la
 *         [5] => cote
 *     )
 */

If we would like to iterate over all the characters, it is recommended to use the IteratorAggregate method, being the Hoa\Ustring\Ustring::getIterator method. Let's see on the Arabic example:

foreach ($arabic as $letter) {
    echo $letter, "\n";
}

/**
 * Will output:
 *     أ
 *     ح
 *     ب
 *     ك
 */

We notice that the iteration is based on the text direction, it means that the first element of the iteration is the first letter of the string starting from the beginning.

Of course, if we would like to get an array of characters, we can use the iterator_to_array PHP function:

print_r(iterator_to_array($arabic));

/**
 * Will output:
 *     Array
 *     (
 *         [0] => أ
 *         [1] => ح
 *         [2] => ب
 *         [3] => ك
 *     )
 */

Comparison and search

Strings can also be compared thanks to the Hoa\Ustring\Ustring::compare method:

$string = new Hoa\Ustring\Ustring('abc');
var_dump(
    $string->compare('wxyz')
);

/**
 * Will output:
 *     string(-1)
 */

This methods returns -1 if the initial string comes before (in the alphabetical order), 0 if it is identical and 1 if it comes after. If we would like to use all the power of the underlying mechanism, we can call the Hoa\Ustring\Ustring::getCollator static method (if the Collator class exists, else Hoa\Ustring\Ustring::compare will use a simple byte to bytes comparison without taking care of the other parameters). Thus, if we would like to sort an array of strings, we will write:

$strings = array('c', 'Σ', 'd', 'x', 'α', 'a');
Hoa\Ustring\Ustring::getCollator()->sort($strings);
print_r($strings);

/**
 * Could output:
 *     Array
 *     (
 *         [0] => a
 *         [1] => c
 *         [2] => d
 *         [3] => x
 *         [4] => α
 *         [5] => Σ
 *     )
 */

Comparison between two strings depends on the locale, it means of the localization of the system, like the language, the country, the region etc. We can use the Hoa\Locale library to modify these data, but it's not a dependence of Hoa\Ustring.

We can also know if a string matches a certain pattern, still expressed with a regular expression. To achieve that, we will use the Hoa\Ustring\Ustring::match method. This method relies on the preg_match and preg_match_all PHP functions, but by modifying the pattern's options to ensure the Unicode support. We have the following parameters: the pattern, a variable passed by reference to collect the matches, flags, an offset and finally a boolean indicating whether the search is global or not (respectively if we have to use preg_match_all or preg_match). By default, the search is not global.

Thus, we will check that our French example contains aime with a direct object complement:

$french->match('#(?:(?<direct_object>\w)[\'\b])aime#', $matches);
var_dump($matches['direct_object']);

/**
 * Will output:
 *     string(1) "t"
 */

This method returns false if an error is raised (for example if the pattern is not correct), 0 if no match has been found, the number of matches else.

Similarly, we can search and replace sub-strings by other sub-strings based on a pattern, still expressed with a regular expression. To achieve that, we will use the Hoa\Ustring\Ustring::replace method. This method uses the preg_replace and preg_replace_callback PHP functions, but still by modifying the pattern's options to ensure the Unicode support. As first argument, we find one or more patterns, as second argument, one or more replacements and as last argument the limit of replacements to apply. If the replacement is a callable, then the preg_replace_callback function will be used.

Thus, we will modify our French example to be more polite:

$french->replace('#(?:\w[\'\b])(?<verb>aime)#', function ($matches) {
    return 'vous ' . $matches['verb'];
});

echo $french;

/**
 * Will output:
 *     Je vous aime
 */

The Hoa\Ustring\Ustring class provides constants which are aliases of existing PHP constants and ensure a better readability of the code:

• Hoa\Ustring\Ustring::WITHOUT_EMPTY, alias of PREG_SPLIT_NO_EMPTY,
• Hoa\Ustring\Ustring::WITH_DELIMITERS, alias of PREG_SPLIT_DELIM_CAPTURE,
• Hoa\Ustring\Ustring::WITH_OFFSET, alias of PREG_OFFSET_CAPTURE and PREG_SPLIT_OFFSET_CAPTURE,
• Hoa\Ustring\Ustring::GROUP_BY_PATTERN, alias of PREG_PATTERN_ORDER,
• Hoa\Ustring\Ustring::GROUP_BY_TUPLE, alias of PREG_SET_ORDER.

Because they are strict aliases, we can write:

$string = new Hoa\Ustring\Ustring('abc1 defg2 hikl3 xyz4');
$string->match(
    '#(\w+)(\d)#',
    $matches,
    Hoa\Ustring\Ustring::WITH_OFFSET
  | Hoa\Ustring\Ustring::GROUP_BY_TUPLE,
    0,
    true
);

Characters

The Hoa\Ustring\Ustring class offers static methods working on a single Unicode character. We have already mentionned the getCharDirection method which allows to know the direction of a character. We also have the getCharWidth which counts the number of columns necessary to print a single character. Thus:

var_dump(
    Hoa\Ustring\Ustring::getCharWidth(Hoa\Ustring\Ustring::fromCode(0x7f)),
    Hoa\Ustring\Ustring::getCharWidth('a'),
    Hoa\Ustring\Ustring::getCharWidth('㽠')
);

/**
 * Will output:
 *     int(-1)
 *     int(1)
 *     int(2)
 */

This method returns -1 or 0 if the character is not printable (for instance, if this is a control character, like 0x7f which corresponds to DELETE), 1 or more if this is a character that can be printed. In our example, 㽠 requires 2 columns to be printed.

To get more semantics, we have the Hoa\Ustring\Ustring::isCharPrintable method which allows to know whether a character is printable or not.

If we would like to count the number of columns necessary for a whole string, we have to use the Hoa\Ustring\Ustring::getWidth method. Thus:

var_dump(
    $french->getWidth(),
    $arabic->getWidth(),
    $japanese->getWidth()
);

/**
 * Will output:
 *     int(9)
 *     int(4)
 *     int(18)
 */

Try this in your terminal with a monospaced font. You will observe that Japanese requires 18 columns to be printed. This measure is very useful if we would like to know the length of a string to position it efficiently.

The getCharWidth method is different of getWidth because it includes control characters. This method is intended to be used, for example, with terminals (please, see the Hoa\Console library).

Finally, if this time we are not interested by Unicode characters but rather by machine characters char (being 1 byte), we have an extra operation. The Hoa\Ustring\Ustring::getBytesLength method will count the length of the string in bytes:

var_dump(
    $arabic->getBytesLength(),
    $japanese->getBytesLength()
);

/**
 * Will output:
 *     int(8)
 *     int(27)
 */

If we compare these results with the ones of the Hoa\Ustring\Ustring::count method, we understand that the Arabic characters are encoded with 2 bytes whereas Japanese characteres are encoded with 3 bytes. We can also get a specific byte thanks to the Hoa\Ustring\Ustring::getByteAt method. Once again, the index is not bounded.

Code-point

Each character is represented by an integer, called a code-point. To get the code-point of a character, we can use the Hoa\Ustring\Ustring::toCode static method, and to get a character based on its code-point, we can use the Hoa\Ustring\Ustring::fromCode static method. We also have the Hoa\Ustring\Ustring::toBinaryCode method which returns the binary representation of a character. Let's take an example:

var_dump(
    Hoa\Ustring\Ustring::toCode('Σ'),
    Hoa\Ustring\Ustring::toBinaryCode('Σ'),
    Hoa\Ustring\Ustring::fromCode(0x1a9)
);

/**
 * Will output:
 *     int(931)
 *     string(32) "1100111010100011"
 *     string(2) "Σ"
 */

Search algorithms

The Hoa\Ustring library provides sophisticated search algorithms on strings through the Hoa\Ustring\Search class.

We will study the Hoa\Ustring\Search::approximated algorithm which searches a sub-string in a string up to k differences (a difference is an addition, a deletion or a modification). Let's take the classical example of a DNA representation: We will search all the sub-strings approximating GATAA with 1 difference (maximum) in CAGATAAGAGAA. So, we will write:

$x      = 'GATAA';
$y      = 'CAGATAAGAGAA';
$k      = 1;
$search = Hoa\Ustring\Search::approximated($y, $x, $k);
$n      = count($search);

echo 'Try to match ', $x, ' in ', $y, ' with at most ', $k, ' difference(s):', "\n";
echo $n, ' match(es) found:', "\n";

foreach ($search as $position) {
    echo '    • ', substr($y, $position['i'], $position['l'), "\n";
}

/**
 * Will output:
 *     Try to match GATAA in CAGATAAGAGAA with at most 1 difference(s):
 *     4 match(es) found:
 *         • AGATA
 *         • GATAA
 *         • ATAAG
 *         • GAGAA
 */

This methods returns an array of arrays. Each sub-array represents a result and contains three indexes: i for the position of the first character (byte) of the result, j for the position of the last character and l for the length of the result (simply j - i). Thus, we can compute the results by using our initial string (here $y) and its indexes.

With our example, we have four results. The first is AGATA, being GATAA with one moved character, and AGATA exists in CAGATAAGAGAA. The second result is GATAA, our sub-string, which well and truly exists in CAGATAAGAGAA. The third result is ATAAG, being GATAA with one moved character, and ATAAG exists in CAGATAAGAGAA. Finally, the last result is GAGAA, being GATAA with one modified character, and GAGAA exists in CAGATAAGAGAA.

Another example, more concrete this time. We will consider the --testIt --foobar --testThat --testAt string (which represents possible options of a command line), and we will search --testot, an option that should have been given by the user. This option does not exist as it is. We will then use our search algorithm with at most 1 difference. Let's see:

$x      = 'testot';
$y      = '--testIt --foobar --testThat --testAt';
$k      = 1;
$search = Hoa\Ustring\Search::approximated($y, $x, $k);
$n      = count($search);

// …

/**
 * Will output:
 *     Try to match testot in --testIt --foobar --testThat --testAt with at most 1 difference(s)
 *     2 match(es) found:
 *         • testIt
 *         • testAt
 */

The testIt and testAt results are true options, so we can suggest them to the user. This is a mechanism user by Hoa\Console to suggest corrections to the user in case of a mistyping.

Conclusion

The Hoa\Ustring library provides facilities to manipulate strings encoded with the Unicode format, but also to make sophisticated search on strings.