Tokens in C and C++ - Hello Codies

The smallest element in C/C++ programs are called TOKENS. These are usually separated by white space like Blanks, Horizontal or vertical tabs, New lines. Tokens in C and C++ recognizes by parser as keywords, variables, constants/literals, identifiers, operators and special symbols etc.

C/C++ Tokens

Keywords
Identifiers
Constants
Variables
Operators
Special symbols

Keywords

Keywords in C/C++ are reserved words as their meaning and purpose already defined in C/C++ libraries used to perform internal operation. Almost all programming language has a set of keywords with predefined values that cannot be used as variable or function names.

Learn more about Keywords in C and C++ at article -C, article – C++.

C has 44 keywords including C89 – 32, C99 – 5, C11 – 7.

C89 Keywords:

         auto         double       int         struct
         break        else         long        switch
         case         enum         register    typedef
         char         extern       return      union
         const        float        short       unsigned
         continue     for          signed      void
         default      goto         sizeof      volatile
         do           if           static      while

C99 Keywords:

        _Bool         _Imaginary   restrict         
        _Complex      inline

C11 Keywords:

       _Alignas        _Atomic       _Noreturn      _Thread_local
       _Alignof        _Generic      _Static_assert

There are 84 keywords in C++ Language including 32 keywords of C language. Below are C++ keywords including C’s too.

//C++98 keywords
and            continue        goto          public             try
and_eq         default         if            register           typedef
asm            delete          inline        reinterpret_cast   typeid
auto           do              int           return             typename
bitand         double          long          short              union
bitor          dynamic_cast    mutable       signed             unsigned
bool           else            namespace     sizeof             using
break          enum            new           static             virtual
case           explicit        not           static_cast        void
catch          export          not_eq        struct             volatile
char           extern          operator      switch             wchar_t
class          false           or            template           while
compl          float           or_eq         this               xor
const          for             private       throw              xor_eq
const_cast     friend          protected     true

//C++11 added keywords

alignas        constexpr       static_assert
alignof        decltype        thread_local  
char16_t       noexcept         
char32_t       nullptr

Above all keywords of both C and C++ are treated as tokens by compilers.

Identifiers

Identifiers refers to the names of entities in a C/C++ program, such as variables, arrays, functions, structures, unions and labels created by user. An identifier is a string of alphanumeric characters that should begin with an alphabet or an underscore character that are used to represent various programming elements such as variables, functions, arrays, structures, unions and so on.

Rules for naming identifiers

An identifier must be begin with an alphabet or an underscore followed only by any number alphabets, digits or underscores.
They should not begin with a digit.
Identifiers are case sensitive i.e. Uppercase and lowercase characters are different, treated as two different entities.
Commas, operators and white spaces not allowed within an identifier.
Keywords cannot be used as an identifier and length cannot be more than 31 characters.

int a, a_b_c; // Valid
int Abs_wer, _123ab; // Valid

int 12abc; // Invalid
int x+y; // Invalid
int +y; // Invalid

Constants

Constants refers to fixed values that cannot be altered during the execution of a program. And You cannot assign values to these terms. Variables which are modifiable can be turned into constant by using keyword const before datatype when you declare a variable.

1, 3.14, x+y are called constant literals to whom you cannot assign value.

Consider the example

int main()
{

// int 1 = 1;  invalid assignment

const float  PI =3.14;	// float constant
const char site_name[]="hellocodies.com"; // string constant

cout<<"PI value :" <<PI<<endl;
cout<<"Site name :" <<site_name<<endl;

// PI = 1;  invalid assignment

	return 0;
}

Output

PI :100    
Site name : hellocodies.com

Variable

Variable is a data storage place with memory allocated to store given different type of input data by user. Variables of different data type requires different amounts of memory and only specific set of operations can be performed on them.

Variables Naming rules

Variable name should not start with a digit.
Special symbols are not allowed in variable name except underscore (_).
Allowed Capital Letters (A – Z), Small Letters (a – z), Digits (0 – 9) and underscore.
Blank or spaces are not allowed within variable name.
Keywords cannot be used as variable name.
C is case-sensitive hence upper and lower case names are different and it is suggested to use variable names in lower case.

Based on Variables availability in or outside block variables are classified as follows.

automatic variable
local variable
global variable
static variable

For more details about variables you can refer this article.

Operators

Operators are that helps us to perform specific mathematical and logical computations on operands. The data variables on which operators act upon are called operands. Depending on the number of operands the operator act upon, operators can be classified as follows:

Unary Operators: Applied only on single operand known as unary operators. For Example, increment, decrement operators.

Binary Operators: Require two operands to perform an operation are called binary operators. All arithmetic operators +, -, *, / etc. are tokens in C/C++ programs. Binary operators are classified into:

Arithmetic operators
Relational Operators
Logical Operators
Bitwise Operators
Assignment Operators
Conditional Operators

Special Symbols

The following special symbols are used in C having some special meaning and thus, cannot be used for some other purpose. [] () {}, ; * = #.

Brackets[]: Opening and closing brackets are used as array element reference and indicates single and multidimensional arrays.

int a[4] = {1, 2, 3, 4}; // 1D array
int b[3][4] = {{1, 2, 3, 4},{5, 6, 7, 8},{9, 10, 11, 12}}; // 2D array

Parentheses(): Indicate function calls and function parameters.
Braces{}: Opening and ending curly braces marks the start and end of a block of code containing more than one statement.
comma ( , ): Comma is used to separate parameters in function calls, variables and also used as operator.

int x=1,y=2; //comma as saperator

int a = (6,4); //comma as operator bcoz a = 4

semi colon( ; ) : Used as statement terminator and also to invokes initialization list in C++.
asterick (*): Used to create pointer variable.
Assignment operator (=): Assign values.
pre processor(#):Used automatically by the compiler to transform your program before actual compilation.