Finding both masses by solving for the acceleration of a two-body gravitational system (instead of using quadratic equation)

1. The usual (quadratic) method

From the force law

$$ m_{1}m_{2}= \frac{Fr^{2}}{G}. $$

Hence the two unknown masses satisfy

$$ x^{2}-Mx+\frac{Fr^{2}}{G}=0 . $$

Solving the quadratic gives

$$ m_{1,2}= \frac{M\pm\sqrt{M^{2}-4\frac{Fr^{2}}{G}}}{2} \approx 0.0594\ \text{kg}\quad\text{and}\quad 3.9406\ \text{kg}. $$

2. What the “system‑acceleration’’ approach actually computes

Treat the whole pair as a single object of mass (M).
Its acceleration (the magnitude of the relative acceleration) is

$$ a=\frac{F}{M}= \frac{2.5\times10^{-10}}{4.00}=6.25\times10^{-11}\ \text{m/s}^{2}. $$

If we now write the gravitational field equation

$$ a=\frac{Gm}{r^{2}}\qquad\Longrightarrow\qquad m=\frac{ar^{2}}{G}, $$

and substitute the expression for (a),

$$ m=\frac{\displaystyle\frac{F}{M}\,r^{2}}{G} =\frac{Fr^{2}}{GM} =\frac{m_{1}m_{2}}{m_{1}+m_{2}} . $$

The quantity

$$ \boxed{\mu\equiv\frac{m_{1}m_{2}}{m_{1}+m_{2}}} $$

is the reduced mass of the two‑body system.
For the numbers in the problem

$$ \mu=\frac{0.2341\ \text{kg}^{2}}{4.00\ \text{kg}} =5.85\times10^{-2}\ \text{kg}\approx0.059\ \text{kg}, $$

which is exactly the smaller mass obtained from the quadratic.

3. Why it coincides with the lighter mass

When one body is much lighter than the other, $(m_{1}<<m_{2}$),

$$ \mu=\frac{m_{1}m_{2}}{m_{1}+m_{2}} =\frac{m_{1}(M-m_{1})}{M} =m_{1}-\frac{m_{1}^{2}}{M} \approx m_{1}, $$

because the correction term $\frac{m_{1}^{2}}{M}$ is only a few percent ($m_{1}/M\approx0.015$ here).
Thus the reduced mass is numerically very close to the lighter mass, which explains the apparent agreement.

4. When the trick does not give an individual mass

If the two masses are comparable, the reduced mass is not equal to either mass. For example, with

$$ m_{1}=m_{2}=2\ \text{kg}, $$

the reduced mass is

$$ \mu=\frac{m_{1}m_{2}}{m_{1}+m_{2}}=1\ \text{kg}, $$

while each actual mass is $2\ \text{kg}$.
Therefore the “system‑acceleration’’ method only yields a useful result when one mass is much smaller than the other; otherwise the quadratic (or an equivalent algebraic system) must be solved.