Lời giải:Đặt \(z=x+yi\text{ }\left( x,\text{ }y\in \mathbb{R} \right)\).
Theo giả thiết, \(\left| z \right|=1\Rightarrow z.\overline{z}=1\) và \({{x}^{2}}+{{y}^{2}}=1\).
\(P=\left| z \right|.\left| {{z}^{2}}-1+2\overline{z} \right|=\left| {{z}^{2}}-1+2\overline{z} \right|=\left| {{x}^{2}}-{{y}^{2}}+2xyi-1+2x-2yi \right|=\left| \left( {{x}^{2}}+2x-{{y}^{2}}-1 \right)+2y\left( x-1 \right)i \right|\)
\(=\sqrt{{{\left( {{x}^{2}}+2x-{{y}^{2}}-1 \right)}^{2}}+4{{y}^{2}}{{\left( x-1 \right)}^{2}}}=\sqrt{{{\left( {{x}^{2}}+2x-1+{{x}^{2}}-1 \right)}^{2}}+4\left( 1-{{x}^{2}} \right){{\left( x-1 \right)}^{2}}}\) (vì \({{y}^{2}}=1-{{x}^{2}}\))
\(=\sqrt{16{{x}^{3}}-4{{x}^{2}}-16x+8}\).
Vì \({{x}^{2}}+{{y}^{2}}=1\Rightarrow {{x}^{2}}=1-{{y}^{2}}\le 1\Rightarrow -1\le x\le 1\).
Xét hàm số \(f\left( x \right)=16{{x}^{3}}-4{{x}^{2}}-16x+8,\text{ }x\in \left[ -1\,;\,1 \right]\).
\({f}'\left( x \right)=48{{x}^{2}}-8x-16\). \({f}'\left( x \right)=0\Leftrightarrow \left[ \begin{align}
& x=-\frac{1}{2}\in \left[ -1\,;\,1 \right] \\
& x=\frac{2}{3}\in \left[ -1\,;\,1 \right] \\
\end{align} \right.\)
\(f\left( -1 \right)=4; f\left( -\frac{1}{2} \right)=13; f\left( \frac{2}{3} \right)=\frac{8}{27}; f\left( 1 \right)=4\).
\(\Rightarrow \underset{\left[ -1\,;\,1 \right]}{\mathop{\max }}\,f\left( x \right)=f\left( -\frac{1}{2} \right)=13\).
Vậy \(\max P=\sqrt{13}\).
