A pet project to re-up my knowledge to C++20/C++23

To re-up to C++23 I needed a project. It can be found here: https://github.com/RichyBoy/hattools. Several years ago I worked out a sorting algorithm and I have now implemented it. Turns out it has been done before (what hasn’t) and it is a pidgeon-hole sort for integers. I will put up a separate post about this project. In truth it is mostly C++20 but I think some of the range stuff I used is from C++23. As I ended up submitting bug reports to gcc I actually built the gcc trunk to take bug fixes for some of the problems I reported..

The Good, The Bad, The Ugly and Dante’s Nine Circles of Hell

Good: Lambdas, Ranges, Concepts, Coroutines and Modules
Bad: Visual Studio has overtaken gcc!
Ugly: Some of the C++20 standard isn’t implemented yet in 2024, never mind C++23.
Inferno: Many, many bugs on gcc affecting even the basics such as module compilation and a lack of co-ordination to facilitate Makefiles properly

A makefile

Although I should move onto CMake I wanted to start with something comfortable - GNU Make. More-or-less everything works as expected in Visual Studio so getting the code to work on gcc was the next challenge.

First comment is that Make works by checking what dependencies have been built already. For C++ modules, that means checking the compiled code that lives in the module cache. The location of this is not exposed by any gcc switch! Remember that Modules are in use to reduce the often colossal and horrendous compile times that can be incurred by large project where a single include could result in hundreds of other includes.. for each file that is built! It is quite suprising that this has been overlooked.

The three variables GCCPATH GCCVERS and CACHE will determine what you need to know.

With modules, you also need to build module partitions first, notice the .ixx postfix. But not before you’ve actually built the header modules, which are no longer being picked up as an include. So the variable MODS list all the modules that import at somepoint in the code, they are built once, and live in the gcm-cache and as already mentioned I’ve worked it so that Make knows about these dependencies.

I couldn’t find anyone out there compiling this stuff with Make so it does make me wonder what kind of adoption C++23 actually has (or C++20 for that matter)

A final note is that I’m building a shared library and then an executable but I have not included the -pthread flag anywhere. I don’t think it is actually needed any more as by default you link to the system library which has long been threaded, but the documentation doesn’t make it clear. I should probably look at that in a bit more detail (build an run a parallel stl::sort and look at the process stack will show the truth) but I think it’s defunct so I’ve ommitted it.

From my project linked above, here is a Makefile:

PROJ 	 = hattools
# gcc doesn't have any options for controlling the gcm.cache location and it is dependant on both the 
# compiler location (prefix) and the compiler version. Or, for GNU Make to check for compiled code in 
# the gcm.cache we need to construct the path of where it lives..
#
# A plea to the gcc gang: please provide something like gcc -dump-gcm.cache 

GCCPATH	:= $(shell which gcc)
GCCVERS := $(shell gcc -dumpversion)
CACHE 	 = gcm.cache/$(subst /bin/gcc,,$(GCCPATH))/include/c++/$(GCCVERS)/
CXXFLAGS = -std=c++2b -fmodules-ts -fconcepts -Wall -O3 -g -fPIC

# C++23 modules for compilation
MODS := chrono \
    vector \
    concepts \
    execution \
    format \
    fstream \
    iostream \
    limits \
    memory \
    random \
    span \
    thread \
    ranges \
    ratio \
    algorithm 

# Project's own module sources, paritions first
SORTSSRC	= sorts/hattools-pidgeon.ixx \
        sorts/hattools-pidgeon23.ixx \
        sorts/hattools-pidgeonflock.ixx \
        sorts/hattools.sorts.ixx		
        #sorts/hattools-radical.ixx 
UTILSSRC	= utils/hattools-generator.ixx \
        utils/hattools-timer.ixx \
        utils/hattools.utils.ixx

# Project's regular source code
SRC	= utils/hattools-timer-impl.cpp 

# Executable source code(s)
EXESRC1	= tests/hatsorttests.cpp
        
DIRS = lib64 bin
TEMP1		:= $(addprefix $(CACHE),$(MODS))
CACHETARGETS	:= $(TEMP1:=.gcm)
SORTSOBJS	= $(SORTSSRC:%.ixx=%.o)
UTILSOBJS	= $(UTILSSRC:%.ixx=%.o)
OBJS		= $(SRC:%.cpp=%.o)
EXEOBJS1	= $(EXESRC1:%.cpp=%.o)
EXE1		= bin/hatsorttests 
LIB		= lib64/lib$(PROJ).so
print-%  : ; @echo $* = $($*) # Courtesy https://www.cmcrossroads.com/article/printing-value-makefile-variable

.PHONY : all 
all : $(shell mkdir -p $(DIRS)) $(CACHETARGETS) $(SORTOBJS) $(UTILSOBJS) $(OBJS) $(LIB) $(EXE1)

exe : $(EXE1)

.PHONY : clean
clean : 
    $(RM) $(SORTSOBJS) $(UTILSOBJS) $(OBJS) $(EXEOBJS1) $(LIB) $(EXE1)
    $(RM) -r gcm.cache $(DIRS)

.PHONY : cacheclean
cacheclean :
    $(RM) -r gcm.cache/usr/*

$(EXE1) : $(CACHETARGETS) $(LIB) $(EXEOBJS1)
    $(CXX) -pie -L$(PWD)/lib64 -Wl,-rpath=$(PWD)/lib64 $(EXEOBJS1) -l$(PROJ) -o $@

$(LIB) : $(SORTSOBJS) $(UTILSOBJS) $(OBJS)
    $(CXX) -shared $^ -o $@ 

%.gcm : 
    $(CXX) $(CXXFLAGS) -x c++-system-header $(subst .gcm,,$(subst $(CACHE),,$@))

%.o : %.ixx
    $(CXX) $(CXXFLAGS) -c -o $@ -x c++ $<

%.o : %.cpp
    $(CXX) $(CXXFLAGS) -c -o $@ $<

Modules and gcc are broken

I submitted 3 different bug reports:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116804
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116803
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=116805

In short, modules break depending on the order of specification and can break depending on various flags. Is anyone actually using this stuff I ask myself ? I think the answer must be “no” because this usage is unavoidable if you use modules. The VS compiler however handled this stuff just fine.

Another ‘issue’ that I didn’t report is below - I am not sure if it has broken something or not although my code worked fine, when I looked at the gcc source code for this I was still unsure - I suspect in my case I’m right on the edge of what gcc maps into memory and if anything was missed out I wasn’t using it anyway. But it is a hard-coded limit which almost certainly shouldn’t be there..

g++ -std=c++2b -fmodules-ts -fconcepts -fcoroutines -Wall -O3 -g -fPIC -c SpaceSort/sorttools-timer-impl.cpp -o SpaceSort/sorttools-timer-impl.o
g++ -shared SpaceSort/sorttools-legacy.o SpaceSort/sorttools-pidgeon.o SpaceSort/sorttools-rapid.o SpaceSort/sorttools.sorts.o SpaceSort/sorttools-generator.o SpaceSort/sorttools-timer.o SpaceSort/sorttools.utils.o SpaceSort/sorttools-timer-impl.o -o SpaceSort/libSpaceSort.so 
g++ -std=c++2b -fmodules-ts -fconcepts -fcoroutines -Wall -O3 -g -fPIC -c SpaceSort/SpaceSort.cpp -o SpaceSort/SpaceSort.o
In module imported at SpaceSort/sorttools-generator.ixx:4:1,
        included from SpaceSort/sorttools.utils.ixx:4,
of module sorttools.utils, imported at SpaceSort/SpaceSort.cpp:12:
/usr/local/include/c++/14.2.0/random: note: unable to represent further imported source locations

Some motivation

I learned C++ at University around 1994/1995 even though it wasn’t on my curricula - I undertook a degree in Electronic Engineering at the University of Southampton so the taught language was C and I recall in ‘96 the Computer Science gang learned Java 1.0 in the Autumn semester. I was happy with C++. I learned the Silicon Graphics Inc (SGI) implementation of the C++ Standard Template Library and then later in the 90s when in industry onto specific implementations such as RogueWave or the native compiler implementations under gcc, on various platforms such as HP-UX, DEC Alpha, Linux etc.

Although I did take an interest in the TR-1 standard, and to an extent Boost I effectively lost interest by the time the C++11 standard came around. Fast-forward a number of years, I’ve decided to re-up my knowledge to the C++23 standard.