Solid-state batteries have attracted intensive attention in recent years for their potential to overcome the limitations of conventional liquid-electrolyte batteries while allowing for compatibility with lithium metal electrodes and improving energy density. On the other hand, there is a growing interest in developing organic electrode materials to supplement existing technologies while offering sustainable, versatile, and potentially low-cost electrochemical storage devices. In this study, we report the first solid-state Li polymer batteries based on poly(ε-caprolactone-co-trimethylene carbonate) 80:20 random copolymer with 28 wt % LiTFSI as the electrolyte paired with high-potential lithiated organic insertion materials, namely, both magnesium and zinc (2,5-dilithium-oxy)-terephthalate compounds. This polymer electrolyte was shown to be compatible with the organic electrode, and functional cells could be constructed. The zinc derivative was selected due to the high ionic potential value of Zn2+, similar to that of Mg2+, and was synthesized in methanol using a scalable and efficient two-step chemical process. The electrochemical study pointed out an optimal operation temperature of 60 °C for such solid-state Li polymer batteries compared with the use of conventional carbonate-based liquid electrolytes at 23 °C. A better stability upon cycling was observed with the magnesium-based phase, while the zinc counterpart showed more pronounced capacity loss, requiring further investigations.