The Internet

John O’Gorman

10 May 2015

1 Intro

1.1 History

1.1.1 Dod and UCB

In response to an initiative of the American Department of Defense (DoD), UCP (University of California Berkeley) developed Unix software and published communication protocols which allowed computers to be connected together using packet switching, From the early 1980s military and university computers were able to transfer files, exchange emails, run terminal sessions on remote machines, and generally communicate with each other in a reliable fashion.

1.1.2 Peers

An important requirement of this DoD project was that nodes on the network would be peers, so that i the event of a nuclear attack (this was during the cold war years) the surviving nodes would still communicate on the network, There was to be no master node, no single weak point, no central node on which all the others depended.

1.1.3 Internet

The software that emerged from Berkeley is now usually called TCP//IP and the network (really a collection of networks) is called the Internet. Commercial and other organisations joined and the internet has exploded from a few thousand nodes in 1983 to about 100 million isn 2002. A whole new glossary of terms came into use to express the concepts on which the Internet is based: interface, packet, protocol, socket, connexion, routing, domain, etc.

1.1.4 Standards, RFCs, etc

In order for the millions of computers on the internet to inter-operate successfully they must conform to common conventions. These conventions are published descriptions of protocols, programming interfaces, file formats, etc promulgated by recognised authorities. There are official standards bodies which are recognised: ANSI (American National Standards Institute), ISO (International Standards Organisation) etc but their progress is glacial and they are largely ignored nowadays. In their place has spring up a system of self-appointed bodies who solicit and promulgate RFCs (Requests for Comment) which become accepted worldwide as de facto standards. The Berkeley TCP/IP and the other internet protocols are in this category.
The official bodies catch up eventually and ratify the de facto RFCs but typically long after each is well established and accepted. An example of a well-respected but unofficial body in the W3C which is an international consortium which promulgates standards for the World Wide Webb (e.g. HTML, XML, etc).

1.2 Key concepts of TCP/IP networking

1.2.1 Hosts and Interfaces

Machines connected to a network are called hosts, Hosts are usually computers, but can be printers, cameras and all sorts of appliances which are being networked these days. They are all called hosts and the combination of software driver and electronic device is called an interface, Interfaces have names like lo0, eth0, ppp0, etc.
lo0 is the loopback interface is usually given an IP address of and the hostname: localhost. The loopback interface is, in principal, a software emulation of an electronic interface and can be used to test network software by simulation a hardware connexion to your local machine.
eth0 is commonly the ethernet interface for the network card in your host machine
ppp0 is the usual name of the point-to-point protocol interface when you have a dialup connexion to your ISP (Internet Service Provider0. This interface uses a serial interface (COM port on PCs) connected to a modem connected you your ISP’s gateway computer. These are regarded as obsolete these days and most ISPs supply a router instead of a modem (but they possibly still call it a modem).
If you have more that 1 instance of any of the above interfaces the 2nd and subsequent instances increment the 0 to 1, 2, and so on (e.g. eth1, eth2, ...)
To see list of all the interfaces configured on your system, type the command:
ifconfig -a

1.2.2 NIC and MAC

Each machine on a network usually has a Network Interface Card or NIC (usually a plug-in card but nowadays often a chipset on the motherboard) which connects to a common wire. The NIC is a transceiver which both receives by detecting voltage flutter on the wire, and transmits by causing similar flutters itself. In concept the wire is a single wire (coaxial thicknet or thinnet) but more commonly now is a UTP (Unshielded Twisted Pair) telephone cable) connexion to a common device (a hub or a switch). All machines on the net hear everything transmitted on the wire. Each NIC in the whole world is manufactures with a unique serial number called a MAC (a 6 byte Machine Address Code value where the 1st 3 bytes identify the manufacturer, and the last 3 bytes identify the card(. Each time the card issues any message it writes its MAC number into the message header. The MAC is usually expressed as a sequence of pairs of hexadecimal digits, colon separated: e.g.: 00;10:5A:34:56:78 is is also known as its hardware address or its ethernet address.
You can see the the MAC number of a NIC on your own computer by running the command: ifconfig
You can see a cache of the MAC numbers of the NICs which have communicated on your network recently. by running the command: arp -a

1.2.3 CSMA/CD

When a NIC sends a message, it listens until no other device is is talking, then begins transmission. While transmitting, it continues to listen, If 2 NICs begin transmission simultaneously, they detect the collision and both stop. Separately they each wait an interval determined by a random number generated then reattempt transmission in the next period of silence. This mechanism is known by the acronym CSMA/CD (Carrier Sense Multiple Access with Collision Detection). The randomness of the wait is crucial to the success (otherwise the devices would continue to interfere with each other forever. To see the number of collisions that have occurred on the NICs on your computer, run the command: ifconfig -a

1.2.4 Packet

Messages are out on the net in packets each of which contains data with a header (and maybe a trailer) attached. The headers have info identifying the source of the messages its intended recipients, and lots of protocol information put there to assist in reliable communication. A packet containing 1 byte of data will typically have a header of 58 or more bytes. Packets usually range from 59 bytes to 1500. They can also, depending on the protocols used, be called segments, datagrams, or frames. Different technologies have varying capacity for packet size (termed MTU, Maximum Transmission unit). To find the MTU value for the various interfaces running on your computer run the command: ifconfig -a

1.2.5 Protocols

The word protocol comes from the world of diplomacy where it refers to a set of standard communication procedures, agreed phraseology, and well defined meanings developed so that international communications may be conducted without misunderstanding.
In the arena of internet standards, the Ps in the alphabet soup of acronyms: ARP, TCP, IP, UDP, ICMP, SMTP, SNTP, FTP, etc all mean protocol. They define how Transport Control, Internet, Universal Datagrams, etc can be effected or communicated without ambiguity between participating machines. Each protocol has an official number, and this protocol number is stored in the package header. The file /etc/protocols contains the list.
To illustrate, the TCP protocol prescribes a 3-way handshake for the establishment of a TCL connexion. Say machine laurel wishes to connect to machine hardy.
  1. laurel sends a SYN n packet (SYN means synchronise and n is a random number chosen by laurel. The number n is the initial sequence number (ISN)).
  2. hardy responds with an ACK n+1, SYN m packet. (ACK means Acknowledge. ACKs always increment the the n sequence number they are acknowledging). m is hardy’s ISN
  3. laurel completes the handshake by sending an ACK m+1 to hardy.
Both machines then listen or talk until one of them sends a FIN packet to the other. The FIN packet (Finish) begins a prescribed 4 way handshake (an ACK followed by an FIN and ACK in the opposite direction). The SYN, ACK, FIN etc are symbolic constants for numbers which are the flag field of the TCP header prepended by TCP to the data it is sending.
You can observe these exchanges of packets by running the command: tcpdump
You need to be superuser root to run this command. The program will display the symbol SYN, ACK etc in its output (which my be voluminous).
The TCP protocol prescribes the values in the flag field of the TCP packet header as:
Flag Value Comment
URG 32 Urgent
ACK 16 Acknowledge
PSH 8 Push, deliver as soon as possible
RST 4 Reset, abort the session
SYN 2 Synchronise, initiate a connexion
FIN 1 Finish sending
Multiple flags can be set in a package, but not all combinations make sense. For any internet protocol, you can find definitions of constants in the .h files in /usr/include/netinet

1.2.6 IP addresses

Every machine on the internet is identified by a 32 bit (4 byte) number called its IP address. This could be expressed as a number between 1 and 4 billion but never is. Instead a decimal dot notation is used e.g. which reveals more readily the location of the host in the hierarchy of addresses.
All hosts with the IP address of the form 192.x.y.z share the high order byte 192. The decimal representation of this range is 402432 to 790527. The decimal dot representation is simply the decimal value of each if the 4 bytes in network order (Big-Endian, most significant byte first). (A byte can have a value between 0 and 255). The numbers you can use are allocated by authorities in bands of values. Typically you can receive them your ISP (Internet Service Provider) who gets them from the national authority (DomainNZ in New Zealand).
Why bother with the IP number when the MAC already identifies the node? Three reasons: ARP
The assignment of an IP address of a device is controlled by the ARP (Address Resolution Protocol). At boot time each machine learns its IP address (in the past from the file /etc/hosts but nowadays more likely from the DHCP (Dynamic Host Control Protocol) and configures its IP address accordingly. Use the commands ifconfig -a on older systems or ip address (on newer systems). The ARP protocol defines a way of broadcasting query packets to learn the MAC of a host with a given IP address.The appropriate device responds and the reply is cached locally for a period of time (typically about 20 minutes). To see a list of cached records, run the command: arp -a RARP
There is also a Reverse Address Resolution Protocol which defines a mechanism for assigning IP address to a device which as a printer interface (e/g/ HP JetDirect) which is not a computer. You read the MAC address on the device then run the command something like:
rarp -s 01:02:3a:03:45:67
The above command will then configure the device with the IP address you have given it.

1.2.7 Net Masks

When an ISP gives you a band of IP address to allocate for machines on your internal network, the allocation is given as a pair. Use the commands ifconfig -a on older systems or ip address (on newer systems). The 2nd element of the pair is a bitmask consisting of a sequence of binary 1s followed by binary zeros. The 1s on the left mark off the network portion which will be common to all nodes on your network. The zeroed portion on the right is the host component and will be unique for each host. In the example below, we have an IP address of with a netmask of 24 bits:
Decimal dot Binary Representation
IP address 11000000 10101000 00000001 00010100
Netmask 11111111 11111111 11111111 00000000
Bitwise AND 11000000 10101000 00000001 00000000
Decimal dot: 192 168 1 0
Note that the leftmost 24 bits of the netmask are all 1s. All the rest to the right are all zeros. All netmasks are like this - some number of bits on the left are 1s, the rest to the right are 0s.
When routing packets around the internet, the NIC performs a bitwise AND of the corresponding bits of both operands, When both bits are 1s, the resulting bit is a 1, otherwise the the resulting bit is a 0. The result is the network IP address with the host component zeroed out.
The above pair would be represented as, the network address(common to all hosts on the network) is 192.168.1 and the 4th number (200 identifies the host. The above pair gives us the capacity to have 254 machines on our 192.168.1 network. We cannot use 0 or 254 as they have a special meaning: 0 means the network, 255 (bits all 1s) is a broadcast address used to communicate simultaneously to all hosts on the network.
An alternative representation gaining in popularity is: where the 24 indicates the number of leftmost 1s in the mask. Routing
The netmask is a key component of routing. Packets destined for hosts on the same network are delivered directly. Hosts are deemed to be on the same network in the process of ANDing their IP addresses with the netmask yields the same value. Hosts on a different network can only be reached via an interface defined as a gateway. To find which (if any) hosts are configured as gateways, run the command netstat -r (or netstat -nr) and look for G in the Flags column.

1.2.8 Network Classes

The original design of the Internet divided the spectrum of IP addresses into classes:
Classes Range Comment
A 1.x.x.x to 126.x.x.x For mega corporations
B 128.x.x.x to 191.255.x.x For large bodies
C 192.0.0.x to 223.255.255.x For the rest of us
The class organisation was dropped in the 1990s in favour of the Network, Netmask divisions described above. The new system is called CIDR (Classless Inter Domain Routing) has allowed greater flexibility of allocation of the address space and filled in some gaps in the utilisation of the IP number space.

1.2.9 Private IP domains

The unforeseen explosion of demand for IP addresses has forced changes to the original scheme, One of the changes specified in RFC 1597, is to declare 3 address domains to be private. They can be used in internal networks and will never be transmitted to the internet. The private domains are:
Network, Netmask Network/Netmask Domain, 10/8 10.x.x.x, 172.16/11 172.16.xx to 172.31.x.x, 192.168/16 192.168.x.x
There can be unlimited replication of private IP domain addresses in remote networks. They cannot collide as they are never seen outside of their own network. This reduces the demand for the public IP addresses (which are a dwindling resource).

1.2.10 IPv6

The use of 32bit IP addresses gave the internet a capacity of 4 billion (232) hosts. In 1980 this seemed plenty (1 for each human in the world). The astounding growth of the internet has forced a rethink. A new standard called IP version 6 or IPv6 has been defined (the version described above is version 4 or IPv4). IPv6 uses 128 bit IP addresses! This increases the capacity of the internet to a mind boggling 34 followed by another 38 digits (2128). This seems to be massive overkill, but it should be remembered there is is an explosion in the use of network devices. Not just computers, but printer devices, surveillance systems, cameral, telephones, and all sorts of domestic appliances being networked.
To represent these new IP numbers, decimal dot notation is replaced by a sequence of 8 hexadecimal numbers separated by colons. Each hexadecimal number is up to 4 digits and represents each 16 bit component of the IPv6 address in network order. Hexadecimal Notation
A bit is a binary digit: 0 or 1. It represents the minimum component of computer memory, storage, or processor registers. It may physically be a transistor switched on or off, or a tiny magnet oriented North-South to South-North. A computer is basically a very large collection of bits which can be flipped by an electronic device called a processor.
The bits of a computer are usually treated in groups of 8 called a byte (BinarY TErm) or more frequently nowadays in official literature an octet. A byte is a sequence of bits with place values which are powers of 2. e.g. the byte 00001010 represents the decimal value 10.
27 26 25 24 23 22 21 20
128 64 32 16 8 4 2 u
0 0 0 0 1 0 1 0
A byte can represent a range of values between 0 (all 0s) and 255 (all 1s). A byte can be used alone to represent a character (each key on the keyboard is assigned a byte value), or they can can be changed together to represent numeric values.
A sequence of 4 bytes (32 bits0 is called a computer word (although this usage is not universal). A word can hold a value of up to 4 billion (232 - 1). Hex
It is common for computer people to represent the values of bytes using base 16 digits. The digits 0 to 9 hare their normal meaning. The letters A to F (or their lower case versions) are used to represent 10 to 15. Each digit represents a power of 16 (in the same way that normal decimal notation represents powers of 10). For example 2a means 2 x 16 + 10 which equals 42 decimal. With this notation, each digit maps easily onto 4 bits (binary digits).
Decimal Hex Binary 4 bits
Digit Place value: 8421
0 0 0000
1 1 0001
2 2 0010
3 3 0011
4 4 0100
5 5 0101
6 6 0110
7 7 0111
8 8 1000
9 9 1001
10 a 1010
11 b 1011
12 c 1100
13 d 1101
14 e 1110
15 f 1111
Using hex notation, a byte can be represented by 2 digits in the range 00 .. ff. This representation is favoured because it is very easy to relate the underlying bit pattern to the digits, in contrast decimal notation conceals the bit pattern of a byte.

1.2.11 IPv6 addresses

Ordinary IPv6 addresses supplied to you by your ISP will start with digit 2 or 3. A random example:
As with IPv4, the netmask will apply to the leftmost bits using the /n notation where n in the number of 1s, and the hosts will be identified by the 128-n bits to the right.
Private IPv6 domains are also mandated. They start with fec0 to feff
Provision is made for collapsing a single run of 4 or more successive zero digits to :: so, for example, ::1 is short for 0:0:0:0:0:0:0:1 and is designated as the IPv6 localhost.
IPv4 IP addresses can be assigned using a hybrid notation. e.g. becomes :: where IPv6 is supported.
Where IPv6 is not supported, the routers will need to convert to a genuine IPv4 address. The notation to show this is ::ffff:
Note that in both the above representations, the leading :: indicates collapsing of zero bytes.

1.2.12 Layers

The Berkeley TCP/IP software model communicates at 4 levels. When an application wishes to send a packet to a remote machine, the packet is formed at top level (the Application layer) and is in turn passed down to the Transport, Internet, and finally the Physical layer as follows
Packets can pass between hosts only at the Physical layer.
When a packet is received at the destination Physical layer, its ethernet card strips off the outer header and trailer and passed the rest of the packet up to the Internet layer. In turn each layer examines and strips of the outermost header before passing the rest of the packet up till it reaches the Application layer. The process is like progressively peeling an onion.
The Transport layer checks the header to ensure that packets have been received in the correct order and reacts appropriately (by sending an ACK or discarding the packet). The sending host with resend any packet which has not been ACKed by the recipient.
Each layer actually communicates ()only with its peer (the implementation of the protocol at the equivalent level on the remote machine). This design means that the modules need not be concerned with details of other layers (they only need the modules to know how to transfer packets to or from the next layer. The modularity of the system makes it easy to add new protocols without disruption to the overall design of the network software. It also means that alternative modules can be used at a layer (e.g. ISDN, ADSL, X25, FDDI, PPP instead of ethernet) without any changes to the other protocol components. OSI layers
The ISO standards body defined a model OSI (Open Systems Interface) with 7 layers compared to TCP.IP;s 4. They expanded the Application into 3L (Session, Presentation, Application), and split the Physical layer into 2: (Physical, Data Link). The OSI is part of a wider definition called GOSIP which was very influential in the 1980s but has declined in importance and is largely ignore today.

1.3 TCP Layer

1.3.1 Multiplexing

There can be many simultaneous connexions between 2 machines. All these connexions must pass over the same wire. The process of merging the packets onto the wire then distributing each to the appropriate remote processes is called multiplexing. The TCP layer l=achieves multiplexing by using ports and sockets to distinguish the multiple strands of connexion.

1.3.2 Ports and Sockets Well known ports
There is a list of 16-bit numbers assigned in the /etc/services file to different network processes. These are called ports. Ports in the range 1 to 1023 are standard, common to all TCP/IP compliant systems, are allocated to services and are described as well known. Some examples: ftp=21, telnet = 23. Registered ports
The file /etc/services also contains ports in the range 1024 to 491511 which are registered with the world authority IANA. Ephemeral ports
Ports above 39151 are not registered and are available as a pool to be used by processes which need them. The highest available number is 65535

1.3.3 Socket

When a TCP/IP connexion is made, each end of the connexion is defined by a pair: IP address and port, which is known as a Berkeley socket. The combination of IP and port must be unique on a host. But there is no need for the combination of IP and port to be identical at each end of the connexion. All packets exchanged through a given socket will be delivered to the socket at the other end of the connexion. only one process can read and write on a socket. Each connexion is fully identified by the 4-tuple of source IP, source port, destination IP, destination port. The IP header contains the IPs and the TCP header the ports. Once created, a socket behaves like a file but without a filename in the directory system, and processes can use normal system calls to red and write on the socket. Unix Domain Sockets
There is a variant of sockets which can be used when the client and server are on the same computer. This implementation creates a pipe which will appear appear with a Unix filename in the file system somewhere. This variant is called a unix Domain socket is twice as fast as the standard form, bypasses the network, and is typically used for local X-Window software.
On Linux, the output of the ls -l command with display a Unix Domain socket with a leading s in the output. To find all the sockets in a Linux system, run the command: find / -type s

1.3.4 Client Server

On end of a connection is deemed to be a server providing a service to the requesting process, the client, at the other end of the connexion. Each Internet server process listens on a well-know port for connexion requests. When a request is received, it is common practice that the server then spawns a child process, passes the connexion to the child which creates a new socket using s port randomly chosen from a pool of ephemeral ports (typically in the range 49152 .. 655635) for its end of the connexion. This technique frees the listener process to accept new requests. The client process also uses a port allocated from the pool of ephemeral ports on its host. To obtain a list of all current connexions, run the command: netstat or netstat -p (The -p option displays the program name).

1.4 Header Layouts

1.4.1 TCP Header

Offset Length Meaning Comment
0 2 Source Port Number
2 2 Destination Port Number
4 4 Sequence Number ISN in SYN flag is set
8 4 ACK Number Valid if ACK flag set
12 2 Header Length + Flags packed 4 bits, 12 bits for flags
14 2 Window Size No of bytes willing to receive
16 2 CRC Checksum
18 2 Urgent pointer valid if URG flag set
20 n= len - 5 Options only if len > 5 else Data
20+n any data

1.4.2 IP Header

Offset Length Field Comment
0 1 Ver&len 4bit version94 or 6), 4 bit header length (words)
1 1 TOS Type of Service (4 flags: see below)
2 2 Length Packet Length in bytes
4 2 ID Serial number (incremented each packet)
6 2 Frag 3-bit flags, 13 bit frag offset
8 1 TTL Time to Live (max hop number, usually 640
9 1 Protocol as /etc/protocols
10 2 CRC Heder Checksum
12 4 SRC Source IP Address
16 4 DST Destination IP address
20 n= len - 5 Options Only if len > 5 else data begin here
20+n any Data
Note that the above layouts are such that 4-byte values always start on 4-byte boundaries (offsets, 0, 4, 8, 12 ...)
The TOS flags (all mutually exclusive) are:
Value Meaning
16 Minimise delay
8 Maximise throughput
4 Maximise Reliability
2 Minimise Monetary Cost
Most systems ignore the TOS flag and it is usually set to 0.

1.4.3 Ethernet header

Offset Length Field Comment
0 6 Source MAC Ethernet Address
6 6 Dest MAC Ethernet Address
12 2 Length bytes of data
14 any Data

1.5 APIs

1.5.1 Berkeley sockets API

The TCP/IP protocols are implemented in the Unix kernel and an Application Programming Interface (API) is provided in the form of system calls or user functions. The basic functions are:
socket() which creates an active socket structure intended for client processes.
connect() which establishes a client connexion to a remote device
bind() which associates a server IP and port (usually well known) with a socket created by socket().
listen() which changes a bound server socket into passive mode where the kernel will pass to it client connexion requests.
accept() which provides a server with a new socket linked to a client request.
connect() and accept() both return file descriptors to the clients and server processes respectively. These file descriptor (really just small integers) are then used for communication using the standard functions: read(), write(),close(), etc.
When writing a client program, a programmer calls socket() and connect().
When writing a server program, a programmer calls socket(); bind(); listen(); then loops calling accept(); typically calling fork(); exec90; to create a new child process to handle each connexion request.
Both server and client use read(); write(); close(); on the sockets just as they would with files or pipes.
For detailed information on the above functions, usr the man or info commands e.g. info socket

1.5.2 The XTI API

There is a competing API, developed at Bell Labs, and intended as an implementation of the OSI Transport Layer Interface (TLI). The Bell Labs TLI was then extended slightly by POSiX (Portable Operating Systems Interface for UniX) and renamed XTI. It offers essentially the same functions as the Berkeley API but prefixes the names with t_ (t_bind(); t_connect();, etc). The arguments to the functions are different, the structures used in the interface differ as well.
Programs using the Berkeley and XTI APS are interoperable - they implement the same protocols: IP, TCP, Ethernet, etc. You can use an XTI client with a Berkeley server, for example.

1.5.3 perl and python sockets API

The programming languages perl and python both implement all the Berkeley functions so that you can create clients and/or servers in these languages