The term mass in special relativity usually refers to the rest mass of the object, which is the Newtonian mass as measured by an observer moving along with the object. The invariant mass is another name for the rest mass of single particles. The more general invariant mass (calculated with a more complicated formula) loosely corresponds to the "rest mass" of a "system." Thus, invariant mass is a natural unit of mass used for systems which are being viewed from their center of momentum frame, as when any closed system (for example a bottle of hot gas) is weighed, which requires that the measurement be taken in the center of momentum frame where the system has no net momentum. Under such circumstances the invariant mass is equal to the relativistic mass (discussed below), which is the total energy of the system divided by c (the speed of light) squared.
The concept of invariant mass does not require bound systems of particles, however. As such, it may also be applied to systems of unbound particles in high-speed relative motion. Because of this, it is often employed in particle physics for systems which consist of widely separated high-energy particles. If such systems were derived from a single particle, calculation of the invariant mass of such systems, a never-changing quantity, gives the rest mass of the parent particle (because it is conserved over time).
Despite the convenience that the invariant mass is the same as the total energy of the system (divided by c2) in the center of momentum frame, the invariant mass of systems (like the rest mass of single particles) is also the same quantity in all inertial frames. Thus, it cannot be destroyed, and is conserved, so long as the system is closed. (In this case, "closure" implies that an idealized boundary is drawn around the system, and no mass/energy is allowed across it).
The term relativistic mass is also sometimes used. This is the sum total quantity of energy in a body or system (divided by c2). As seen from the center of momentum frame, the relativistic mass is also the invariant mass, as discussed above (just as the relativistic energy of a single particle is the same as its rest energy, when seen from its rest frame). For other frames, the relativistic mass (of a body or system of bodies) includes a contribution from the "net" kinetic energy of the body (the kinetic energy of the center of mass of the body), and is larger the faster the body moves. Thus, unlike the invariant mass, the relativistic mass depends on the observer's frame of reference. However, for given single frames of reference and for closed systems, the relativistic mass is also a conserved quantity.
Although some authors present relativistic mass as a fundamental concept of the theory, it has been argued that this wrong as the fundamentals of the theory relate to space-time. There is disagreement over whether the concept is pedagogically useful.    The notion of mass as a property of an object from Newtonian mechanics does not bear a precise relationship to the concept in relativity.
If a stationary box contains many particles, it weighs more in its rest frame, the faster the particles are moving. Any energy in the box (including the kinetic energy of the particles) adds to the mass, so that the relative motion of the particles contributes to the mass of the box. But if the box itself is moving (its center of mass is moving), there remains the question of whether the kinetic energy of the overall motion should be included in the mass of the system. The invariant mass is calculated excluding the kinetic energy of the system as a whole (calculated using the single velocity of the box, which is to say the velocity of the box's center of mass), while the relativistic mass is calculated including invariant mass PLUS the kinetic energy of the system which is calculated from the velocity of the center of mass.
Relativistic mass and rest mass are both traditional concepts in physics, but the relativistic mass corresponds to the total energy. The relativistic mass is the mass of the system as it would be measured on a scale, but in some cases (such as the box above) this fact remains true only because the system on average must be at rest to be weighed (it must have zero net momentum, which is to say, the measurement is in its center of momentum frame). For example, if an electron in a cyclotron is moving in circles with a relativistic velocity, the weight of the cyclotron+electron system is increased by the relativistic mass of the electron, not by the electron's rest mass. But the same is also true of any closed system, such as an electron-and-box, if the electron bounces at high speed inside the box. It is only the lack of momentum in the system which allows the kinetic energy of the electron to be "weighed." If the electron is stopped and weighed, or the scale were somehow sent after it, it would not be moving with respect to the scale, and again the relativistic and rest masses would be the same for the single electron (and would be smaller). In general, relativistic and rest masses are equal only in systems which have no net momentum and the system center of mass is at rest; otherwise they may be different.
The invariant mass is proportional to the value of the total energy in one reference frame, the frame where the object as a whole is at rest (as defined below in terms of center of mass). This is why the invariant mass is the same as the rest mass for single particles. However, the invariant mass also represents the measured mass when the center of mass is at rest for systems of many particles. This special frame where this occurs is also called the center of momentum frame, and is defined as the inertial frame in which the center of mass of the object is at rest (another way of stating this is that it is the frame in which the momenta of the system's parts add to zero). For compound objects (made of many smaller objects, some of which may be moving) and sets of unbound objects (some of which may also be moving), only the center of mass of the system is required to be at rest, for the object's relativistic mass to be equal to its rest mass.
A so-called massless particle (such as a photon, or a theoretical graviton) moves at the speed of light in every frame of reference. In this case there is no transformation that will bring the particle to rest. The total energy of such particles becomes smaller and smaller in frames which move faster and faster in the same direction. As such, they have no rest mass, because they can never be measured in a frame where they are at rest. This property of having no rest mass is what causes these particles to be termed "massless."
It was recognized by J. J. Thomson in 1881 that a charged body is harder to set in motion than an uncharged body, which was worked out on more detail by Oliver Heaviside (1889) and George Frederick Charles Searle (1897).  
So the electrostatic energy behaves as having some sort of electromagnetic mass, which can increase the normal mechanical mass of the bodies. Later Wilhelm Wien (1900), Max Abraham (1902), came to the conclusion that the total mass of the bodies is identical to its electromagnetic mass. And because the em-mass depends on the em-energy, the formula for the energy-mass-relation given by Wien (1900) was m = (4 / 3)E / c2.  
It was pointed out by Thomson and Searle, that this electromagnetic mass also increases with velocity. This was also recognized by Hendrik Lorentz (1899, 1904) in the framework of Lorentz's Theory of Electrons. He defined mass as the ratio of force to acceleration not as the ratio of momentum to velocity, so he needed to distinguish between the mass mL = γ3m0 parallel to the direction of motion and the mass mT = γm0 perpendicular to the direction of motion. Only when the force is perpendicular to the velocity is Lorentz's mass equal to what is now called "relativistic mass". (Where is the Lorentz factor, v is the relative velocity between the aether and the object, and c is the speed of light). Abraham (1902) called mL longitudinal mass and mT transverse mass, (whereby Abraham's own expressions were more complicated than Lorentz's relativistic ones). So, according to this theory no body can reach the speed of light because the mass becomes infinitely large at this velocity.  
The precise relativistic expression (which is equivalent to Lorentz's) relating force and acceleration for a particle with non-zero rest mass m moving in the x direction with velocity v and associated Lorentz factor γ is
Einstein calculated the longitudinal and transverse mass (which are equivalent to those of Lorentz, but for a mistake in mT, which was later corrected ) in his 1905 electrodynamics paper and in another paper in 1906.   However, in his first paper on E = mc2 (1905) he treated m as what would now be called the rest mass.  Some claim that (in later years) he did not like the idea of "relativistic mass": 
It is not good to introduce the concept of the mass of a moving body for which no clear definition can be given. It is better to introduce no other mass concept than the ’rest mass’ m. Instead of introducing M it is better to mention the expression for the momentum and energy of a body in motion.
– Albert Einstein in letter to Lincoln Barnett, 19 June 1948 (quote from L. B. Okun, “The Concept of Mass,” Phys. Today 42, 31, June 1989.)
In special relativity, an object that has a mass cannot travel at the speed of light. As the object approaches the speed of light, the object's energy and momentum increase without bound.
The velocity dependent mass of Lorentz and Abraham were replaced by the concept of relativistic mass, an expression which was first defined by Richard C. Tolman in 1912, who stated: “the expression m0(1 - v2/c2)-1/2 is best suited for THE mass of a moving body.”
In 1934, Tolman also defined relativistic mass as
which holds for all particles, including those moving at the speed of light.
For a slower than light particle, a particle with a nonzero rest mass, the formula becomes
Tolman remarked on this relation that "We have, moreover, of course the experimental verification of the expression in the case of moving electrons to which we shall call attention in §29. We shall hence have no hesitation in accepting the expression as correct in general for the mass of a moving particle."
When the relative velocity is zero, γ is simply equal to 1, and the relativistic mass is reduced to the rest mass as one can see in the next two equations below. As the velocity increases toward the speed of light c, the denominator of the right side approaches zero, and consequently γ approaches infinity.
In the formula for momentum
the mass that occurs is the relativistic mass. In other words, the relativistic mass is the proportionality constant between the velocity and the momentum.
Newton's second law remains valid in the form
the derived form is not valid because in is generally not a constant  (see the section above on transverse and longitudinal mass).
Many contemporary authors such as Taylor and Wheeler avoid using the concept of relativistic mass altogether:
The relativistic expressions for E and p obey the relativistic energy-momentum equation:
the m is the rest mass.
The equation is also valid for photons, which have m=0:
a photon's momentum is a function of its energy, but it is not proportional to the velocity, which is always c.
For an object at rest, the momentum p is zero,
And the rest mass is only equal to the total energy in the rest frame of the object.
If the object is moving, the total energy is
Which has both positive and negative solutions. In classical physics, the negative energy solutions are spurious, and as the momentum increases with the increase of the velocity v, so does the total energy.
To find the form of the momentum and energy as a function of velocity, note that the four-velocity, which is proportional to , is the only four-dimensional arrow associated to the particle's motion, so that if there is a conserved four-momentum , it must be proportional to this vector. This gives the ratio of energy and momentum:
Which makes the energy-momentum equation a relation between E and v.
Which gives E
The relativistic mass equation is the formula for E divided by c2
The equation is often written this way because the difference E2 − p2 is the relativistic length of the energy momentum four-vector. In the rest frame, where p = 0, the equation just states that E=m, again revealing that the rest mass is the energy in the rest frame.
The rest mass of a composite system is not the sum of the rest masses of the parts, unless all the parts are at rest. The total mass of a composite system includes the kinetic energy and field energy in the system.
The total energy E of a composite system can be determined by adding together the sum of the energies of its components. The total momentum of the system, a vector quantity, can also be computed by adding together the momenta of all its components. Given the total energy E and the length (magnitude) p of the total momentum vector , the invariant mass is given by:
In a mathematical system where c = 1, for systems of particles (whether bound or unbound) the total system invariant mass is given equivalently by the following:
Where, again, the particle momenta are first summed as vectors, and then the square of their resulting total magnitude (Euclidean norm) is used. This results in a scalar number, which is subtracted from the scalar value of the square of the total energy.
For such a system, in the special center of momentum frame where momenta sum to zero, again the system mass (called the invariant mass) corresponds to the total system energy or, in units where c=1, is identical to it. This invariant mass for a system remains the same quantity in any inertial frame, although the system total energy and total momenta are functions of the particular inertial frame which is chosen, and will vary in such a way between inertial frames as to keep the invariant mass the same for all observers. Invariant mass thus functions for systems of particles in the same capacity as "rest mass" does for single particles.
Note that the invariant mass of a closed system is also independent of observer or inertial frame, and is a constant, conserved quantity for closed systems and single observers, even during chemical and nuclear reactions. It is widely used in particle physics, because the invariant mass of a particle's decay products is equal to its rest mass. This is used to make measurements of the mass of particles like the Z boson or the top quark.
Total energy is an additive conserved quantity (for single observers) in systems and in reactions between particles, but rest mass (in the sense of being a sum of particle rest masses) may not be conserved through an event in which rest masses of particles are converted to other types of energy, such as kinetic energy. Finding the sum of individual particle rest masses would require multiple observers, one for each particle rest inertial frame, and these observers ignore individual particle kinetic energy. Conservation laws require a single observer and a single inertial frame.
In general, for closed systems and single observers, relativistic mass is conserved (each observer sees it constant over time), but is not invariant (that is, different observers see different values). Invariant mass, however, is both conserved and invariant (all single observers see the same value, which does not change over time).
The relativistic mass corresponds to the energy, so conservation of energy automatically means that relativistic mass is conserved for any given observer and inertial frame. However, this quantity, like the total energy of a particle, is not invariant. This means that, even though it is conserved for any observer during a reaction, its absolute value will change with the frame of the observer, and for different observers in different frames.
By contrast, the rest mass and invariant masses of systems and particles are both conserved and also invariant. For example: A closed container of gas has a system "rest mass" in the sense that it can be weighed on a resting scale, even while it contains moving components. This mass is the invariant mass, which is equal to the total relativistic energy of the container (including the kinetic energy of the gas) only when it is measured in the center of momentum frame. Just as is the case for single particles, the calculated "rest mass" of such a container of gas does not change when it is in motion, although its relativistic mass does.
The container may even be subjected to a force which gives it an over-all velocity, or else (equivalently) it may be viewed from an inertial frame in which it has an over-all velocity (that is, technically, a frame in which its center of mass has a velocity). In this case, its total relativistic mass and energy increase. However, in such a situation, although the container's total relativistic energy and total momenta increase, these energy and momentum increases subtract out in the invariant mass definition, so that the moving container's invariant mass will be calculated as the same value as if it were measured at rest, on a scale.
All conservation laws in special relativity (for energy, mass, and momentum) require closed systems, meaning systems that are totally isolated, with no mass-energy allowed in or out, over time. If a system is closed/isolated, then both total energy and total momentum in the system are conserved over time for any observer in any single inertial frame, though their absolute values will vary, according to different observers in different inertial frames. The invariant mass of the system is also conserved, but does not change with different observers. This is also the familiar situation with single particles: all observers calculate the same particle rest mass (a special case of the invariant mass) no matter how they move (what inertial frame they choose), but different observers see different total energies and momenta for the same particle.
Conservation of invariant mass also requires the system to be enclosed so that no heat and radiation (and thus invariant mass) can escape. As in the example above, a physically enclosed or bound system does not need to be completely isolated from external forces for its mass to remain constant, because for bound systems these merely act to change the inertial frame of the system or the observer. Though such actions may change the total energy or momentum of the bound system, these two changes cancel, so that there is no change in the system's invariant mass. This is just the same result as with single particles: their calculated rest mass also remains constant no matter how fast they move, or how fast an observer sees them move.
On the other hand, for systems which are unbound, the "closure" of the system may be enforced by an idealized surface, inasmuch as no mass-energy can be allowed into or out of the test-volume over time, if conservation of system invariant mass is to hold during that time. If a force is allowed to act on (do work on) only one part of such an unbound system, this is equivalent to allowing energy into or out of the system, and the condition of "closure" to mass-energy is violated. In this case, conservation of invariant mass of the system also will no longer hold.
Again, in special relativity, the rest mass of a system is not required to be equal to the sum of the rest masses of the parts (a situation which would be analogous to gross mass-conservation in chemistry). For example, a massive particle can decay into photons which individually have no mass, but which (as a system) preserve the invariant mass of the particle which produced them. Also a box of moving non-interacting particles (e.g., photons, or an ideal gas) will have a larger invariant mass than the sum of the rest masses of the particles which compose it. This is because the total energy of all particles and fields in a system must be summed, and this quantity, as seen in the center of momentum frame, and divided by c2, is the system's invariant mass.
In special relativity, mass is not "converted" to energy, for all types of energy still retain their associated mass. Neither energy nor invariant mass can be destroyed in special relativity, and each is separately conserved over time in closed systems. Thus, a system's invariant mass may change only because invariant mass is allowed to escape, perhaps as light or heat. Thus, when reactions (whether chemical or nuclear) release energy in the form of heat and light, if the heat and light is not allowed to escape (the system is closed and isolated), the energy will continue to contribute to the system rest mass, and the system mass will not change. Only if the energy is released to the environment will the mass be lost; this is because the associated mass has been allowed out of the system, where it contributes to the mass of the surroundings.
According to Lev Okun, Einstein himself always meant the invariant mass when he wrote "m" in his equations, and never used an unqualified "m" symbol for any other kind of mass. Okun and followers reject the concept of relativistic mass. Arnold B. Arons has argued against teaching the concept of relativistic mass:
For many years it was conventional to enter the discussion of dynamics through derivation of the relativistic mass, that is the mass–velocity relation, and this is probably still the dominant mode in textbooks. More recently, however, it has been increasingly recognized that relativistic mass is a troublesome and dubious concept. [See, for example, Okun (1989).]... The sound and rigorous approach to relativistic dynamics is through direct development of that expression for momentum that ensures conservation of momentum in all frames:
rather than through relativistic mass....
On the other hand, T. R. Sandin has written:
The concept of relativistic mass brings a consistency and simplicity to the teaching of special relativity to introductory students. For example, E = mc2 then expresses the beautifully simplifying equivalence of mass and energy. Those who claim not to use relativistic mass actually do so—if not by name—when considering systems of particles or photons. Relativistic mass does not depend on the angle between force and velocity—this supposed dependence results from incorrect use of Newton's second law of motion.