Boy, I sure have a lot of links I can drop
about this subject!
It's really a super broad topic and involves a lot of areas. Penta's two bits, mostly formed from reading a bunch of other things people have said...1) Start with the battle system and know what it can do.
I mean beyond visual effects like making the sprites poof around and smack each other. Does your battle system come with a reflect mechanic? A specific way of deciding turns? Things like that! Know everything you have to work with, and it really helps to find a central "idea" or "theme" you can build your battle system around. To indulge in myself a bit, one of the Pentalen's current projects is based entirely around mastering an elemental system. I'm bringing it up now because it's something I can easily reference when trying to talk about strategy -- I know it inside out since I made it, ah-hahaha.
And don't think you're limited by what battle system you use! If you want to invent a new system, work within the confines of whatever engine you're using. Even the DBS can be tweaked into something new. To go back to my self-indulgent example, the big custom thing there is the ability to chain together skills to perform damaging combos. It's important to note that the skill combos are based on elements, tying itself to the theme of "master an elemental system".2) Once you know what's in your toolbox, build your characters!
When you're making your characters, it's generally a good idea to build them around archetypes -- the fighter, the tank, the healer, things like that. It gives the player something they can easily understand and figure out how to use, but don't stop there! Try making different versions within the usual archetypes
, or even making your own classes. Using them right out of the can is very predictable, and while that can work for some games, it can reduce the strategy in your game to "get a tank and a healer and someone to blow up enemies", or something really basic.3) Once you have characters, make your enemies.
I recommend fitting most enemies into archetypes, too. Stick them in troops that allow them to work well together. (An enemy that can heal in a group of fighter-type enemies, but the fighter-type enemies would protect the healer with their own support abilities.) Don't just slap abilities onto them, or make them able to inflict a variety of status effects! Build them a bit like how you would make players, though really there are so many ways to approach designing enemies that it's a topic and a half all on its own.4) Allow the player to actually have choices.
I would say this is one of the most important things to remember, if not the most important. You don't create strategy by pulling things like "Well the big hairy wolf is weak to having its fur set on fire, so just do that for a bit and boom, it's dead." That just changes the "spam attack" problem to "spam fire". There's no choice about it, you'll pick the fire ability for as long as you can if you know the enemy is weak to fire. This kind of strategy really works well as an introduction to an elemental system, however. A rather drawn-out example, but...
Say you're in the Noob Cave and you run into a bunch of slimes. They're all painfully weak to fire and just melt in one hit if you use it. This makes any battles with the slimes very easy, which is great for your garden-variety battles -- not everything needs to be a 10 turn struggle of epic proportions. Then you get to the boss, and it's a giant slime. Most players will use a fire skill on it right off the bat, but what's this? It's such a big pile of goo that the fire isn't much more effective than a regular attack! Now, if you were a nice developer, you'd have dropped hints of some kind that hey, maybe ice would be useful here. So the player uses an ice skill, but... that's not entirely effective, either! However, now the slime could be sensitive to fire spells, and while this would prompt the player to spam as much fire as they could while the boss was vulnerable, it's a lot more rewarding than just using a single ability the entire battle right off the bat.
There are a lot of things you can do to encourage choices, and a lot of it depends on how each part of your battle system relates to the next. Take status effects: by tying them to something like elements, suddenly it might be a good idea to use that darkness spell on an enemy that won't take much damage on it, because darkness can inflict blindness, and the enemy in question is a heavy-hitting physical attacker. Alternatively you could try to paralyze him with a thunder spell, or expend resources in an attempt to inflict as much damage as possible so you could kill the bastard quickly.
Tl;dr: THERE ARE A LOT OF THINGS YOU CAN DO. VERY FEW ARE CERTAIN GUARANTEES, HOWEVER. Also slavery